Unicode Tutorials - Herong's Tutorial Examples - v5.32, by Herong Yang
Unicode Character Encoding and Decoding
This section provides tutorial example on how to convert Unicode character strings to and from byte sequences based on different encoding standards using str.encode() and bytes.decode() methods.
Python support a large number of encodings that convert Unicode character strings into byte sequences according to different human language coding standards. Of course, Python allows you to convert those byte sequences back to Unicode characters strings.
str.encode(encoding='utf-8', errors='strict') - encode() is an instance method of a "str" object that converts the Unicode code points represented by the "str" object into a byte sequence according to the given encoding standard.
bytes.decode(encoding='utf-8', errors='strict') - decode() is an instance method of a "bytes" object that converts the byte sequence to a character string according to the given encoding standard.
Here are names of some encoding standards as described earlier in this book.
Here is a Python script that shows you how to use str.encode() and bytes.decode() methods. The "replace" error handling option is used is used to avoid conversion exceptions.
# Character-Encoding-Decoding.py
# Copyright 2019 (c) HerongYang.com. All Rights Reserved.
#
en = "English"
fr = "Français"
zh = "中文"
print("Original character strings:")
print(" {0}, {1}, {2}".format(en, fr, zh))
print("Encoded with default UTF-8 encoding:")
enBin = en.encode()
frBin = fr.encode()
zhBin = zh.encode()
print(" {0}, {1}, {2}".format(enBin, frBin, zhBin))
print("Decoded back to Character strings:")
enStr = enBin.decode()
frStr = frBin.decode()
zhStr = zhBin.decode()
print(" {0}, {1}, {2}".format(enStr, frStr, zhStr))
print("Encoded and decoded back with Latin-1 encoding:")
enBin = en.encode("latin-1")
frBin = fr.encode("latin-1")
zhBin = zh.encode("latin-1", "replace")
enStr = enBin.decode("latin-1")
frStr = frBin.decode("latin-1")
zhStr = zhBin.decode("latin-1")
print(" {0}, {1}, {2}".format(enBin, frBin, zhBin))
print(" {0}, {1}, {2}".format(enStr, frStr, zhStr))
print("Encoded and decoded back with GB2312 encoding:")
enBin = en.encode("gb2312")
frBin = fr.encode("gb2312", "replace")
zhBin = zh.encode("gb2312")
enStr = enBin.decode("gb2312")
frStr = frBin.decode("gb2312")
zhStr = zhBin.decode("gb2312")
print(" {0}, {1}, {2}".format(enBin, frBin, zhBin))
print(" {0}, {1}, {2}".format(enStr, frStr, zhStr))
Run the above script, it will print the following output:
herong$ python3 Character-Encoding-Decoding.py Original character strings: English, Français, 中文 Encoded with default UTF-8 encoding: b'English', b'Fran\xc3\xa7ais', b'\xe4\xb8\xad\xe6\x96\x87' Decoded back to Character strings: English, Français, 中文 Encoded and decoded back with Latin-1 encoding: b'English', b'Fran\xe7ais', b'??' English, Français, ?? Encoded and decoded back with GB2312 encoding: b'English', b'Fran?ais', b'\xd6\xd0\xce\xc4' English, Fran?ais, 中文
The output confirms that Python does support encoding standards for different human languages. It has a large database that maps Unicode characters to codes defined in each individual encoding standard.
Table of Contents
ASCII Character Set and Encoding
GB2312 Character Set and Encoding
GB18030 Character Set and Encoding
JIS X0208 Character Set and Encodings
UTF-8 (Unicode Transformation Format - 8-Bit)
UTF-16, UTF-16BE and UTF-16LE Encodings
UTF-32, UTF-32BE and UTF-32LE Encodings
►Python Language and Unicode Characters
Summary of Unicode Support in Python
Unicode Support on "str" Data Type
►Unicode Character Encoding and Decoding
"unicodedata" Module for Unicode Properties
Java Language and Unicode Characters
Encoding Conversion Programs for Encoded Text Files
Using Notepad as a Unicode Text Editor
Using Microsoft Word as a Unicode Text Editor