Unicode Tutorials - Herong's Tutorial Examples - v5.32, by Herong Yang
Features of UTF-8 Encoding
This section provides a quick summary of features of UTF-8 encoding. UTF-8 is very efficient for Western language characters.
Features of UTF-8 encoding:
1.UTF-8 is very efficient in storage for characters in Western languages. The code points for most of characters in Western languages are in the U+0000...U+007F range, which will be encoded with 1 byte per character. The rest of characters will be encoded by two bytes. Some reports says that any text data in Western language will be encoded with 1.1 bytes per character on average. This is a big saving comparing with any other Unicode encodings.
2. UTF-8 is compatible with the single byte ASCII encoding. In another word, ASCII encoding can be viewed as a sub set of the UTF-8 encoding. Any ASCII byte stream is a valid UTF-8 byte stream.
3. UTF-8 is backward compatible with Unicode 3.0 character set, which only contains characters in the U+0000...U+FFFF range.
4. UTF-8 is less efficient in storage for characters in CJK (Chinese, Japanese, and Korean) languages comparing to UTF-16 encoding. The code points for most of characters in CJK languages are in the U+000800...U+00FFFF range, which will be encoded in UTF-8 with 3 bytes per character. UTF-16 will encode all characters 2 bytes per character.
5. Processing a UTF-8 encoded text files is relatively easy. If the leading bit of the first byte is a 0, then 1 byte is used to encoding this character. If the leading bit of the first byte is not a 0, then the number of non-zero leading bits is total number bytes used to encoding this character.
6. Another nice nature of UTF-8 encoding is that all subsequent bytes in encoded multi-byte sequence are having the pattern of 0b10xxxxxx in binary format. If you are looking at one byte of an encoded character in the middle of the encoded stream, and want to find out the first byte of this encoded character, you just need to follow this simple logic:
while (current byte matches the bit pattern '10xxxxxx') {
Current byte = previous byte
}
Table of Contents
ASCII Character Set and Encoding
GB2312 Character Set and Encoding
GB18030 Character Set and Encoding
JIS X0208 Character Set and Encodings
►UTF-8 (Unicode Transformation Format - 8-Bit)
UTF-16, UTF-16BE and UTF-16LE Encodings
UTF-32, UTF-32BE and UTF-32LE Encodings
Python Language and Unicode Characters
Java Language and Unicode Characters
Encoding Conversion Programs for Encoded Text Files
Using Notepad as a Unicode Text Editor
Using Microsoft Word as a Unicode Text Editor