Root Cause of Corrupted Chinese Text

∟Chinese Text Encoding Conversion and Corruptions

∟Root Cause of Corrupted Chinese Text

This section provides a tutorial example to demonstrate the root cause of corrupted Chinese text - incorrect 8-bit encodings are used to decode original Chinese text.

Based on my experience, the root cause of corrupted Chinese text can be divided into the following 3 encoding processing mistakes:

1. The original Chinese text is encoded in UTF-8, but it gets decoded as one of the 8-bit encodings.

2. The original Chinese text is encoded in Unicode (UTF-16BE), but it gets decoded as one of the 8-bit encodings.

3. The original Chinese text is encoded in GB18030 (GBK), but it gets decoded as one of the 8-bit encodings.

There are many 8-bit encodings. And here are some examples:

ISO-8859-1 or Latin-1 used as default for many systems. The x'80' - x'FF' range covers characters for Western European languages.
ISO-8859-15 (or Revised Latin-1). The x'80' - x'FF' range gets revised.
Windows-1252 or CP1252 used as default for older Windows systems. The x'80' - x'FF' range extends ISO-8859-1.
Extended ASCII or CP437 (IBM437) used as default for IBM-PC DOS systems and other hardware. The x'80' - x'FF' range covers some symbols, some Western European language characters, and some box drawing characters.
Extended ASCII for Latin (Revised) or CP852. The x'80' - x'FF' range gets designed for Latin characters.
Extended ASCII for Japanese or CP932 The x'80' - x'FF' range gets designed for Japanese Kanji characters.

With 3 possible encodings in the original text and many possible decoding mistaken options, the resulting corrupted Chinese text will be a large number of variations. The following PHP script shows 12 of possible corrupted output from a single Chinese text:

<?php 
#- Chinese-Corrupted-Encoding.php
#- Copyright (c) 2005 HerongYang.com. All Rights Reserved.

  # Original in Unicode (UTF-16BE)
  $original = hex2bin('7b804f534e2d65877f519875');
  corrupted($original, "UTF-16BE");
  
  # Original in UTF-8
  $original = hex2bin('e7ae80e4bd93e4b8ade69687e7bd91e9a1b5');
  corrupted($original, "UTF-8");
  
  # Original in GB18030 (GBK)
  $original = hex2bin('bcf2cce5d6d0cec4cdf8d2b3');
  corrupted($original, "GB18030");

function corrupted($original, $encoding) {
  print("\nOriginal encoding: ".$encoding."$\n");
  print("   Text: ".iconv($encoding, "UTF-8", $original)."$\n");
  print("   Binary: ".bin2hex($original)."$\n");
  print("   Corrupted as:\n");
  print("      ISO-8859-1: "
    .iconv("ISO-8859-1", "UTF-8//IGNORE", $original)."$\n");
  print("      CP437: ".iconv("CP437", "UTF-8//IGNORE", $original)."$\n");
  print("      CP852: ".iconv("CP852", "UTF-8//IGNORE", $original)."$\n");
  print("      CP932: ".iconv("CP932", "UTF-8//IGNORE", $original)."$\n");
}
?>