FromUCStoUTF.pptVIP

From UCS-2 to UTF-16 Discussion and practical example for the transition of a Unicode library from UCS-2 to UTF-16 Why is this an issue? The concept of the Unicode standard changed during its first few years Unicode 2.0 (1996) expanded the code point range from 64k to 1.1M APIs and libraries need to follow this change and support the full range Upcoming character assignments (Unicode 3.1, 2001) fall into the added range “Unicode is a 16-bit character set” Concept: 16-bit, fixed-width character set Saving space by not including precomposed, rarely-used, obsolete, … characters Compatibility, transition strategies, and acceptance forced loosening of these principles Unicode 3.1: 90k assigned characters 16-bit APIs APIs developed for Unicode 1.1 used 16-bit characters and strings: UCS-2 Assuming 1:1 character:code unit Examples: Win32, Java, COM, ICU, Qt/KDE Byte-based UTF-8 (1993) mostly for MBCS compatibility and transfer protocols Extending the range Set aside two blocks of 1k 16-bit values, “surrogates”, for extension 1k x 1k = 1M =additional code points using a pair of code units 16-bit form now variable-width UTF-16 “Unicode scalar values” 0..10ffff16 Proposed: 1994; part of Unicode 2.0 (1996) Parallel with ISO-10646 ISO-10646 uses 31-bit codes: UCS-4 UCS-2: 16-bit codes for subset 0..ffff16 UTF-16: transformation of subset 0..10ffff16 UTF-8 covers all 31 bits Private Use areas above 10ffff16 slated for removal from ISO-10646 for UTF interoperability and synchronization with Unicode 21-bit code points Code points (“Unicode scalar values”) up to 10ffff16 use 21 bits 16-bit code units still good for strings: variable-width like MBCS Default string unit size not big enough for code points Dual types for programming? C: char/wchar_t dual types C/C++ standards: dual types Strings mostly with char units (8 bits) Code points: wchar_t, 8..32 bits Typical use in I18N-ed programs: (8-bit) char strings but (16/32-bit) wchar_t (or 32-bit int) characters; code point