Character Set Encoding Map Analyzer

This section provides a tutorial example on how to write a simple program to analyze and print out the encoding map showing relations between character code points and their encoded byte sequences of a given encoding.

As mentioned in the previous chapter, Java 11 supports 171 built-in character set encodings.

In order to figure out the encoding map (relations between character code points and their encoded byte sequences) of a specific supported encoding, I wrote the following program to analyze a given encoding and print a map between code points (from 0x0000 to 0x10FFFF) and their encoded byte sequences:

/* EncodingAnalyzer2.java
 * Copyright (c) 2019 HerongYang.com. All Rights Reserved.
 */
import java.io.*;
class EncodingAnalyzer2 {
   static char hexDigit[] = {'0', '1', '2', '3', '4', '5', '6', '7',
                             '8', '9', 'A', 'B', 'C', 'D', 'E', 'F'};
   public static void main(String[] a) {
      String charset = null;
      if (a.length>0) charset = a[0];
      if (charset==null) System.out.println("Default encoding:");
      else System.out.println(charset+" encoding:");
      int lastByte = 0;
      int lastLength = 0;
      byte[] startSequence = null;
      int startChar = 0;
      byte[] endSequence = null;
      int endChar = 0;
      boolean isFirstChar = true;
      for (int i=0; i<=0x10FFFF; i++) {
         int c = i;
         String s = new String(Character.toChars(c));
         byte[] b = null;
         if (charset==null) {
            b = s.getBytes();
         } else {
            try {
               b = s.getBytes(charset);
            } catch (UnsupportedEncodingException e) {
               System.out.println(e.toString());
               break;
            }
         }
         int l = b.length;
         int lb = ((int) b[l-1]) & 0x00FF;
         if (isFirstChar==true) {
            isFirstChar = false;
            startSequence = b;
            startChar = c;
            lastByte = lb - 1;
            lastLength = l;
         }
         if (!(l==lastLength && (lb==lastByte+1 || lb==lastByte))) {
            System.out.print(intToHex(startChar)+" >");
            printBytes(startSequence);
            System.out.print(" - "+intToHex(endChar)+" >");
            printBytes(endSequence);
            if (lastByte==0x3F) System.out.print(": Invalid range");
            System.out.println("");
            startSequence = b;
            startChar = c;
         }
         endSequence = b;
         endChar = c;
         lastLength = l;
         lastByte = lb;
      }
      System.out.print(intToHex(startChar)+" >");
      printBytes(startSequence);
      System.out.print(" - "+intToHex(endChar)+" >");
      printBytes(endSequence);
      if (lastByte==0x3F) System.out.print(": Invalid range");
      System.out.println("");
   }
   public static void printBytes(byte[] b) {
      for (int j=0; j<b.length; j++)
         System.out.print(" "+byteToHex(b[j]));
   }   
   public static String byteToHex(byte b) {
      char[] a = { hexDigit[(b >> 4) & 0x0f], hexDigit[b & 0x0f] };
      return new String(a);
   }
   public static String charToHex(char c) {
      byte hi = (byte) (c >>> 8);
      byte lo = (byte) (c & 0xff);
      return byteToHex(hi) + byteToHex(lo);
   }
   public static String intToHex(int i) {
      char hi = (char) (i >>> 16);
      char lo = (char) (i & 0xffff);
      return charToHex(hi) + charToHex(lo);
   }
}

Note that:

The output of this program will be discussed in sections bellow.

Table of Contents

 About This Book

 Character Sets and Encodings

 ASCII Character Set and Encoding

 GB2312 Character Set and Encoding

 GB18030 Character Set and Encoding

 JIS X0208 Character Set and Encodings

 Unicode Character Set

 UTF-8 (Unicode Transformation Format - 8-Bit)

 UTF-16, UTF-16BE and UTF-16LE Encodings

 UTF-32, UTF-32BE and UTF-32LE Encodings

 Python Language and Unicode Characters

 Java Language and Unicode Characters

 Character Encoding in Java

Character Set Encoding Maps

Character Set Encoding Map Analyzer

 Character Set Encoding Maps - US-ASCII and ISO-8859-1/Latin 1

 Character Set Encoding Maps - CP1252/Windows-1252

 Character Set Encoding Maps - Unicode UTF-8

 Character Set Encoding Maps - Unicode UTF-16, UTF-16BE, UTF-16LE

 Character Set Encoding Maps - Unicode UTF-32, UTF-32BE, UTF-32LE

 Character Counter Program for Any Given Encoding

 Character Set Encoding Comparison

 Encoding Conversion Programs for Encoded Text Files

 Using Notepad as a Unicode Text Editor

 Using Microsoft Word as a Unicode Text Editor

 Using Microsoft Excel as a Unicode Text Editor

 Unicode Fonts

 Archived Tutorials

 References

 Full Version in PDF/EPUB