Standard Compression Scheme for Unicode
Encyclopedia
The Standard Compression Scheme for Unicode (SCSU) is a Unicode
Technical Standard for reducing the number of byte
s needed to represent Unicode text, especially if that text uses mostly characters from one or a small number of per-language character blocks. It does so by dynamically mapping values in the range 128–255 to offsets within particular blocks of 128 characters. The initial conditions of the encoder mean that existing strings in ASCII
and ISO-8859-1 that do not contain C0 control codes other than NULL TAB CR and LF can be treated as SCSU strings. Since most alphabets do reside in blocks of contiguous Unicode codepoints, texts that use small alphabets and either ASCII punctuation or punctuation that fits within the window for the main alphabet can be encoded at one byte per character (plus setup overhead, which for common languages is often only 1 byte), most other punctuation can be encoded at 2 bytes per symbol through non-locking shifts. SCSU can also switch to UTF-16 internally to handle non-alphabetic languages.
Symbian OS, an operating system for mobile phones and other mobile devices, uses SCSU to serialise strings.
Reuters
, the organization that floated the first draft of SCSU, is believed to use SCSU internally.
SQL Server 2008 R2
uses SCSU to compress Unicode values stored in nchar(n) and nvarchar(n) columns, achieving space savings between 15% and 50%, depending on the language of the data.
Treated purely as a compression algorithm, SCSU is inferior to most commonly-used general-purpose algorithms for texts of over a few kilobytes. One of several problems with SCSU is then that the savings of SCSU versus UTF-16 or UTF-8
drop after external compression, often dramatically so.
SCSU does have the advantage that it can usefully compress texts that are only a few characters long, whereas most full-scale compressors need hundreds of bytes of data to break even against their own overhead. In Symbian OS, SCSU is used even for Clipboard operations, e.g. Cut, Copy & Paste of small strings of text.
Unicode
Unicode is a computing industry standard for the consistent encoding, representation and handling of text expressed in most of the world's writing systems...
Technical Standard for reducing the number of byte
Byte
The byte is a unit of digital information in computing and telecommunications that most commonly consists of eight bits. Historically, a byte was the number of bits used to encode a single character of text in a computer and for this reason it is the basic addressable element in many computer...
s needed to represent Unicode text, especially if that text uses mostly characters from one or a small number of per-language character blocks. It does so by dynamically mapping values in the range 128–255 to offsets within particular blocks of 128 characters. The initial conditions of the encoder mean that existing strings in ASCII
ASCII
The American Standard Code for Information Interchange is a character-encoding scheme based on the ordering of the English alphabet. ASCII codes represent text in computers, communications equipment, and other devices that use text...
and ISO-8859-1 that do not contain C0 control codes other than NULL TAB CR and LF can be treated as SCSU strings. Since most alphabets do reside in blocks of contiguous Unicode codepoints, texts that use small alphabets and either ASCII punctuation or punctuation that fits within the window for the main alphabet can be encoded at one byte per character (plus setup overhead, which for common languages is often only 1 byte), most other punctuation can be encoded at 2 bytes per symbol through non-locking shifts. SCSU can also switch to UTF-16 internally to handle non-alphabetic languages.
Symbian OS, an operating system for mobile phones and other mobile devices, uses SCSU to serialise strings.
Reuters
Reuters
Reuters is a news agency headquartered in New York City. Until 2008 the Reuters news agency formed part of a British independent company, Reuters Group plc, which was also a provider of financial market data...
, the organization that floated the first draft of SCSU, is believed to use SCSU internally.
SQL Server 2008 R2
Microsoft SQL Server
Microsoft SQL Server is a relational database server, developed by Microsoft: It is a software product whose primary function is to store and retrieve data as requested by other software applications, be it those on the same computer or those running on another computer across a network...
uses SCSU to compress Unicode values stored in nchar(n) and nvarchar(n) columns, achieving space savings between 15% and 50%, depending on the language of the data.
Comparison with external compression schemes
SCSU has not been a resounding success. Few applications need to compress so much Unicode text that it's worth using a special-purpose compression scheme which does not have widespread support. Also, while it can be used as a text encoding, it can be difficult to handle internally.Treated purely as a compression algorithm, SCSU is inferior to most commonly-used general-purpose algorithms for texts of over a few kilobytes. One of several problems with SCSU is then that the savings of SCSU versus UTF-16 or UTF-8
UTF-8
UTF-8 is a multibyte character encoding for Unicode. Like UTF-16 and UTF-32, UTF-8 can represent every character in the Unicode character set. Unlike them, it is backward-compatible with ASCII and avoids the complications of endianness and byte order marks...
drop after external compression, often dramatically so.
SCSU does have the advantage that it can usefully compress texts that are only a few characters long, whereas most full-scale compressors need hundreds of bytes of data to break even against their own overhead. In Symbian OS, SCSU is used even for Clipboard operations, e.g. Cut, Copy & Paste of small strings of text.
See also
- Binary Ordered Compression for UnicodeBinary Ordered Compression for UnicodeBinary Ordered Compression for Unicode is a MIME compatible Unicode compression scheme. BOCU-1 combines the wide applicability of UTF-8 with the compactness of Standard Compression Scheme for Unicode . This Unicode encoding is designed to be useful for compressing short strings, and maintains code...
(BOCU-1) - International Components for UnicodeInternational Components for UnicodeInternational Components for Unicode is an open source project of mature C/C++ and Java libraries for Unicode support, software internationalization and software globalization. ICU is widely portable to many operating systems and environments. It gives applications the same results on all...
A library that can convert between SCSU and other Unicode encodings