Unicode Tutorials - Herong's Tutorial Examples

∟GB18030 Character Set and Encoding

∟GB18030 Encoding for GB18030 Character Set

This section provides a quick introduction of the GB18030 encoding for the GB18030 character set. GB18030 is a multi-byte (1-byte, 2-byte, or 4-byte) encoding.

GB18030 encoding scheme uses one, two or four bytes to encode a character. The following table shows the ranges of valid byte sequences: 

Number Of   Valid Range
Bytes       Byte 1        Byte 2        Byte 3        Byte 4

   1        0x00 - 0x7F
   2        0x81 - 0xFE   0x40 - 0x7E
   2        0x81 - 0xFE   0x80 - 0xFE
   4        0x81 - 0xFE   0x30 - 0x39   0x81 - 0xFE   0x30 - 0x39

Processing a GB18030 encoded byte stream from the beginning of the stream is easy. Here is an algorithm to divide the stream into sequences of bytes that represent valid GB18030 characters: 

Input: 
   byte stream in
Algorithm:
   while (in.hasNext())
      b1 = in.nextByte()
      if (0x00 <= b1 && b1 <= 0x7F)
         b1 is a valid byte sequence
      else if (0x81 <= b1 && b1 <= 0xFE && in.hasNext())
         b2 = in.nextByte()
         if ((0x40 <= b2 && b2 <= 0x7E) || (0x80 <= b2 && b2 <= 0xFE))
            b1, b2 is a valid byte sequence
         else if (0x30 <= b2 && b2 <= 0x39 && in.hasNext())
            b3 = in.nextByte()
            if (0x81 <= b3 && b3 <= 0xFE && in.hasNext())
               b4 = in.nextByte()
               if (0x30 <= b4 && b4 <= 0x39)
                  b1, b2, b3, b4 is a valid byte sequence
               else
                  stream is corrupted
               end if
            else 
               stream is corrupted
            end if
         else 
            stream is corrupted
         end if
      else 
         stream is corrupted
      end if
   end while

Thanks to Mark, who provided suggestions to make the above algorithm more accurate.

Table of Contents

 About This Book

 Character Sets and Encodings

 ASCII Character Set and Encoding

 GB2312 Character Set and Encoding

►GB18030 Character Set and Encoding

 History of GB Character Sets

►GB18030 Encoding for GB18030 Character Set

 JIS X0208 Character Set and Encodings

 Unicode Character Set

 UTF-8 (Unicode Transformation Format - 8-Bit)

 UTF-16, UTF-16BE and UTF-16LE Encodings

 UTF-32, UTF-32BE and UTF-32LE Encodings

 Python Language and Unicode Characters

 Java Language and Unicode Characters

 Character Encoding in Java

 Character Set Encoding Maps

 Encoding Conversion Programs for Encoded Text Files

 Using Notepad as a Unicode Text Editor

 Using Microsoft Word as a Unicode Text Editor

 Using Microsoft Excel as a Unicode Text Editor

 Unicode Fonts

 Archived Tutorials

 References

 Full Version in PDF/EPUB