Unicode Character Encoding and Decoding

Unicode Tutorials - Herong's Tutorial Examples

∟Unicode Character Encoding and Decoding

This section provides tutorial example on how to convert Unicode character strings to and from byte sequences based on different encoding standards using str.encode() and bytes.decode() methods.

Python support a large number of encodings that convert Unicode character strings into byte sequences according to different human language coding standards. Of course, Python allows you to convert those byte sequences back to Unicode characters strings.

str.encode(encoding='utf-8', errors='strict') - encode() is an instance method of a "str" object that converts the Unicode code points represented by the "str" object into a byte sequence according to the given encoding standard.

bytes.decode(encoding='utf-8', errors='strict') - decode() is an instance method of a "bytes" object that converts the byte sequence to a character string according to the given encoding standard.

Here are names of some encoding standards as described earlier in this book.

"ascii" for the ASCII (American Standard Code for Information Interchange) encoding standard covering 128 characters including English letters and numbers.
"latin-1" for the Latin-1, also called ISO 8859-1, standard covering 256 characters including letters for Western European languages.
"gb2312" for the GB2312 standard covering basic Chinese characters.
"utf-8" for the UTF-8 standard covering all Unicode characters.
"utf-16" for the UTF-16 standard covering all Unicode characters.
"utf-32" for the UTF-32 standard covering all Unicode characters.

Here is a Python script that shows you how to use str.encode() and bytes.decode() methods. The "replace" error handling option is used is used to avoid conversion exceptions.

# Character-Encoding-Decoding.py
# Copyright 2019 (c) HerongYang.com. All Rights Reserved.
#
en = "English"
fr = "Français"
zh = "中文"

print("Original character strings:")
print("   {0}, {1}, {2}".format(en, fr, zh))

print("Encoded with default UTF-8 encoding:")
enBin = en.encode()
frBin = fr.encode()
zhBin = zh.encode()
print("   {0}, {1}, {2}".format(enBin, frBin, zhBin))

print("Decoded back to Character strings:")
enStr = enBin.decode()
frStr = frBin.decode()
zhStr = zhBin.decode()
print("   {0}, {1}, {2}".format(enStr, frStr, zhStr))

print("Encoded and decoded back with Latin-1 encoding:")
enBin = en.encode("latin-1")
frBin = fr.encode("latin-1")
zhBin = zh.encode("latin-1", "replace")
enStr = enBin.decode("latin-1")
frStr = frBin.decode("latin-1")
zhStr = zhBin.decode("latin-1")
print("   {0}, {1}, {2}".format(enBin, frBin, zhBin))
print("   {0}, {1}, {2}".format(enStr, frStr, zhStr))

print("Encoded and decoded back with GB2312 encoding:")
enBin = en.encode("gb2312")
frBin = fr.encode("gb2312", "replace")
zhBin = zh.encode("gb2312")
enStr = enBin.decode("gb2312")
frStr = frBin.decode("gb2312")
zhStr = zhBin.decode("gb2312")
print("   {0}, {1}, {2}".format(enBin, frBin, zhBin))
print("   {0}, {1}, {2}".format(enStr, frStr, zhStr))

Run the above script, it will print the following output:

herong$ python3 Character-Encoding-Decoding.py 

Original character strings:
   English, Français, 中文

Encoded with default UTF-8 encoding:
   b'English', b'Fran\xc3\xa7ais', b'\xe4\xb8\xad\xe6\x96\x87'
Decoded back to Character strings:
   English, Français, 中文

Encoded and decoded back with Latin-1 encoding:
   b'English', b'Fran\xe7ais', b'??'
   English, Français, ??

Encoded and decoded back with GB2312 encoding:
   b'English', b'Fran?ais', b'\xd6\xd0\xce\xc4'
   English, Fran?ais, 中文

The output confirms that Python does support encoding standards for different human languages. It has a large database that maps Unicode characters to codes defined in each individual encoding standard.