Character Counter Program for Any Given Encoding

This section provides a tutorial example on how to write a simple program to count valid characters in a give encoding character set encoding.

As mentioned in the previous chapter, Java SE 7 supports 166 built-in character set encodings.

Of course, each encoding is designed for a specific character set only. As a simple exercise, I want to write a sample program that counts the number of characters in the character set of a given encoding.

The sample program, EncodingCounter.java, counts the number of code points that are mapped valid byte sequences in the 0x0000 - 0x10FFFF range for a given encoding:

/**
 * EncodingCounter2.java
 - Copyright (c) 2012, HerongYang.com, All Rights Reserved.
 */
import java.io.*;
import java.nio.*;
import java.nio.charset.*;
class EncodingCounter2 {
   static char hexDigit[] = {'0', '1', '2', '3', '4', '5', '6', '7',
                             '8', '9', 'A', 'B', 'C', 'D', 'E', 'F'};
   public static void main(String[] a) {
      String charset = "CP1252";
      if (a.length>0) charset = a[0];
      System.out.println(charset+" encoding:");
      int lastByte = 0;
      int lastLength = 0;
      byte[] startSequence = null;
      int startChar = 0;
      byte[] endSequence = null;
      int endChar = 0;
      boolean isFirstChar = true;
      int validCount = 0;
      int subCount = 0;
      int totalCount = 0x00110000;
      for (int i=0; i<totalCount; i++) {
         subCount++;
         int c = i;
         byte[] b = encodeByEncoder(c,charset);         
         int l = 0;
         int lb = 0;
         if (b!=null) {
            l = b.length;
            lb = ((int) b[l-1]) & 0x00FF;
            validCount++;
         }
         if (isFirstChar==true) {
            isFirstChar = false;
            startSequence = b;
            startChar = c;
            lastByte = lb - 1;
            lastLength = l;
         }
         if (!(l==lastLength)) {
            System.out.print(intToHex(startChar)+" >");
            printBytes(startSequence);
            System.out.print(" - "+intToHex(endChar)+" >");
            printBytes(endSequence);
            System.out.println(" = "+(subCount-1));
            startSequence = b;
            startChar = c;
            subCount = 1;
         }
         endSequence = b;
         endChar = c;
         lastLength = l;
         lastByte = lb;
      }
      System.out.print(intToHex(startChar)+" >");
      printBytes(startSequence);
      System.out.print(" - "+intToHex(endChar)+" >");
      printBytes(endSequence);
      System.out.println(" = "+(subCount));
      System.out.println("Total characters = "+totalCount);
      System.out.println("Valid characters = "+validCount);
      System.out.println("Invalid characters = "
         +(totalCount-validCount));
   }
   public static byte[] encodeByEncoder(int c, String cs) {
      Charset cso = null;
      byte[] b = null;
      try {   	
         cso = Charset.forName(cs);
         CharsetEncoder e =  cso.newEncoder();
         e.reset();
         ByteBuffer bb
            = e.encode(CharBuffer.wrap(Character.toChars(c)));
         if (bb.limit()>0) b = copyBytes(bb.array(),bb.limit());
      } catch (IllegalCharsetNameException e) {
         System.out.println(e.toString());
      } catch (CharacterCodingException e) {
         // invalid character, return null
      }      	
      return b;
   }
   public static void printBytes(byte[] b) {
      if (b!=null) {
         for (int j=0; j<b.length; j++)
            System.out.print(" "+byteToHex(b[j]));
      } else {
         System.out.print(" XX");
      }
   }   
   public static byte[] copyBytes(byte[] a, int l) {
      byte[] b = new byte[l];
      for (int i=0; i<Math.min(l,a.length); i++) b[i] = a[i];
      return b;
   }
   public static String byteToHex(byte b) {
      char[] a = { hexDigit[(b >> 4) & 0x0f], hexDigit[b & 0x0f] };
      return new String(a);
   }
   public static String charToHex(char c) {
      byte hi = (byte) (c >>> 8);
      byte lo = (byte) (c & 0xff);
      return byteToHex(hi) + byteToHex(lo);
   }
   public static String intToHex(int i) {
      char hi = (char) (i >>> 16);
      char lo = (char) (i & 0xffff);
      return charToHex(hi) + charToHex(lo);
   }
}

Note that:

The output of this program will be discussed in the next section.

Last update: 2012.

Table of Contents

 About This Book

 Character Sets and Encodings

 ASCII Character Set and Encoding

 GB2312 Character Set and Encoding

 GB18030 Character Set and Encoding

 JIS X0208 Character Set and Encodings

 Unicode Character Set

 UTF-8 (Unicode Transformation Format - 8-Bit)

 UTF-16, UTF-16BE and UTF-16LE Encodings

 UTF-32, UTF-32BE and UTF-32LE Encodings

 Java Language and Unicode Characters

 Character Encoding in Java

Character Set Encoding Maps

 Character Set Encoding Map Analyzer

 Character Set Encoding Maps - US-ASCII and ISO-8859-1/Latin 1

 Character Set Encoding Maps - CP1252/Windows-1252

 Character Set Encoding Maps - Unicode UTF-8

 Character Set Encoding Maps - Unicode UTF-16, UTF-16BE, UTF-16LE

 Character Set Encoding Maps - Unicode UTF-32, UTF-32BE, UTF-32LE

Character Counter Program for Any Given Encoding

 Character Set Encoding Comparison

 Encoding Conversion Programs for Encoded Text Files

 Using Notepad as a Unicode Text Editor

 Using Microsoft Word as a Unicode Text Editor

 Using Microsoft Excel as a Unicode Text Editor

 Unicode Fonts

 Unicode Code Point Blocks - Code Charts

 Outdated Tutorials

 References

 PDF Printing Version