Bytes, encodings, and Unicode

What you'll learn

• Tell str (text) and bytes (raw data) apart, and convert between them
• Explain what an encoding is and why UTF-8 is the default you should always name
• Recognize and fix the two classic failures: UnicodeDecodeError and mojibake
• Handle files and network data that arrive as bytes

Concept

A string in Python is a sequence of characters: letters, digits, emoji, Bengali script, anything in Unicode. A file on disk or data on a network is a sequence of bytes: numbers from 0 to 255. Something has to translate between the two, and that something is an encoding. "café".encode("utf-8") turns four characters into five bytes; .decode("utf-8") turns them back. Text is what you work with; bytes are what gets stored and sent.

UTF-8 is the encoding the whole modern world agreed on. It stores plain English letters in one byte each, so files that only use them are identical to the old ASCII files, and it stores everything else in two to four bytes. Every website, every JSON API, and every source file assumes it. The reason to write encoding="utf-8" explicitly is that Python's default depends on the operating system's settings, and a file written on one machine can then fail to open on another. Naming the encoding makes your code behave the same everywhere.

Two failures come from mismatched encodings. A UnicodeDecodeError means the bytes could not be interpreted with the encoding you named: usually a file written in a legacy encoding like Latin-1 being read as UTF-8. Mojibake is the quieter one: the decode succeeds with the wrong table, and café appears as café. When you see that pattern, the file was UTF-8 but was read as Latin-1 or Windows-1252.

You will meet bytes directly in three places: files opened in binary mode with "rb" or "wb", which is how images and other non-text files are handled; data from network sockets and HTTP bodies, which arrive as bytes and must be decoded; and the len() of encoded text, which is what actually counts against storage and transfer limits. A byte string literal is written b"..." and only allows ASCII characters inside the quotes.

Code

Encoding and decoding, with the lengths that show why the two types must not be confused.

# encode_decode.py
text = "café 🐍"
data = text.encode("utf-8")

print(type(text), len(text))
print(type(data), len(data))
print(data)
print(data.decode("utf-8"))
print(data.decode("utf-8") == text)

<class 'str'> 6
<class 'bytes'> 10
b'caf\xc3\xa9 \xf0\x9f\x90\x8d'
café 🐍
True

The bytes each character becomes. hex() shows the byte values listed in the table earlier.

# per_char.py
for ch in "Aéক🐍":
    encoded = ch.encode("utf-8")
    print(f"{ch!r:6} U+{ord(ch):04X}  {encoded.hex(' ')}  ({len(encoded)} bytes)")

'A'    U+0041  41  (1 bytes)
'é'    U+00E9  c3 a9  (2 bytes)
'ক'    U+0995  e0 a6 95  (3 bytes)
'🐍'    U+1F40D  f0 9f 90 8d  (4 bytes)

Both failure modes, produced on purpose. The first decode uses the wrong table and quietly returns mojibake; the second raises because the bytes cannot be UTF-8.

# mismatch.py
utf8_bytes = "café".encode("utf-8")
print(utf8_bytes.decode("latin-1"))

latin1_bytes = "café".encode("latin-1")
print(latin1_bytes)
print(latin1_bytes.decode("utf-8"))

café
b'caf\xe9'
Traceback (most recent call last):
  File "mismatch.py", line 6, in <module>
    print(latin1_bytes.decode("utf-8"))
          ~~~~~~~~~~~~~~~~~~~^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 3: unexpected end of data

Files in binary mode, and the explicit-encoding habit that prevents cross-platform surprises.

# binary_files.py
from pathlib import Path

Path("greeting.txt").write_text("héllo\n", encoding="utf-8")

with open("greeting.txt", "rb") as f:
    raw = f.read()
print(raw, len(raw), "bytes on disk")

with open("greeting.txt", encoding="utf-8") as f:
    text = f.read()
print(repr(text), len(text), "characters")

with open("copy.bin", "wb") as f:
    f.write(raw)
print(Path("copy.bin").read_bytes() == raw)

b'h\xc3\xa9llo\n' 7 bytes on disk
'héllo\n' 6 characters
True

Try it yourself

Create encodings.py. Take a sentence with at least one accented letter and one emoji. Print its character count and its UTF-8 byte count, then its byte count in "utf-16". Write it to a file with encoding="utf-8", read the file back in binary mode, and decode the bytes yourself to prove they match. Finally, decode the same bytes as "latin-1" and look at the mojibake.

Common mistakes

• Treating bytes as text. b"abc" + "def" is a TypeError; decode first, then work with strings.
• Leaving the encoding implicit. open(path) without encoding= uses a platform default that differs between Windows and everything else; always name it.
• Fixing mojibake by replacing characters. café is a symptom of decoding with the wrong table; fix the decode, not the string.

What's next

• str is characters, bytes is numbers 0 to 255; encode and decode translate using an encoding table.
• UTF-8 is the universal default; always pass encoding="utf-8" so code behaves the same on every machine.
• UnicodeDecodeError means the wrong table for the bytes; mojibake means a wrong table that happened to succeed.

You know what text really is under the surface. Next you will learn to handle every kind of failure properly, with exceptions: catching them, raising them, and designing your own.