The text encoding error field guide

Text encodings are everywhere, invisible in most cases but ever present. While most text is encoded as UTF-8 nowadays, computer environments don't always treat it as a first class citizen. This article will give you an overview of the problems around text encoding and common error scenarios, intended as a pointer to when and how such issues manifest, and how to debug them in the real world.

A brief overview of text encodings

Modern text encodings started with the standardization of ASCII, a 7 bit character set assigning the 128 most common english characters and control sequences to numeric values. Using only 7 bits might seem wasteful from a modern perspective, but at the time a byte was not standardized to contain 8 bits, and systems that did use 8 bit bytes used the leftover bit for parity (think a primitive checksum for error detection).

Then computing gained traction outside of the english speaking world, where using only 128 characters is very limited as soon as you introduce regional character variations like é or symbols like €. To combat the problem, ISO standard 8859 introduced several new charsets that used 8 bits for storage and extended the previous ASCII values by 128 regional characters, symbols and (unprintable) control codes. While ISO 8859 defines 15 different charsets for different regions, the by far most popular was ISO 8859-1, also known as latin-1, covering extra symbols for the western european region, where most commercial computing industry was concentrated at the time.

A little later, Microsoft decided to publish the Windows 1252 charset, basically latin-1 but with the control codes replaced by less common printable characters. There is some confusion around the latin-1 charset because of this, and a file encoded as latin-1 on windows may be using the Windows 1252 charset instead of ISO 8859-1 internally, which are not fully compatible with each other despite their shared base.

The asian continent had it even worse, because even their most common alphabet wouldn't fit into a single byte, so they developed entirely different multi-byte encodings, where multiple bytes represented a single character, for example Shift_JIS for japanese or GBK for chinese.

All this mess is obviously unsustainable in a globalized economy, with software services offered in every country and most languages, causing the creation of unicode - a standardized alphabet large enough to contain every single region-specific character, symbol and control sequence. There are 3 primary text encodings using the unicode alphabet: UTF-32 using 4 bytes for every character, UTF-16 using 2 bytes for the most common ones and 4 bytes for the rest, and UTF-8 using 1-4 bytes per character.

UTF-8 is the standard on virtually all modern systems because it uses the least amount of bytes to store english text out of all unicode charsets, avoids the byte order confusion of UTF-16/UTF-32 and most importantly it is fully ASCII-compatible, so technically every ASCII text file is valid UTF-8 and any UTF-8 text using only ASCII characters is also valid ASCII. This backward-compatibility allowed configuration and source code files, that stuck with ASCII throughout the encoding mess, to gracefully upgrade without changing a single byte.

Mojibake

When encoding text in one charset and then decoding it as a different one, the resulting garbage output is called mojibake. In theory, any incompatible combination of charsets will produce it, but today you almost always produce it by accidentally feeding unicode to a program supporting only latin-1 or ASCII, like databases or terminal tools.

If you want to try it yourself, here is a python one-liner to reproduce it:

print("Café".encode("utf-8").decode("latin-1", errors="replace"))

For example, decoding the UTF-8 string Café as latin-1 would show:

Café

UTF-16 and UTF-32 are easier to catch, since they start with a Byte-Order-Mark (BOM) to define endianness and byte order. A UTF16/32 string Café decoded as latin-1 would display:

ÿþCafé

The first two characters ÿþ are the little endian BOM bytes FF FE (and the additional 00 00 for UTF32, but those are not printable).

Mojibake errors are most obvious because they typically do not affect the ASCII character range, but garbles anything else. Fixing it is fairly simple: figure out the correct charset, and decode as that.

On linux, you can check the charset of a file with:

file -i sample.txt

The output will display the filename, mime type and detected charset:

sample.txt: text/plain; charset=utf-8

If the charset is not compatible with your target program, you can convert it, for example from latin-1 (ISO 8859-1) to UTF-8:

iconv -f ISO-8859-1 -t UTF-8 sample_latin1.txt > sample_utf8.txt

Databases

Database administrators may decide to use the fixed 1 byte latin-1 charset instead of the variable length utf-8 as the internal text encoding, because it is easier to parse and takes up less (or at worst equal) storage space, while still offering a decent alphabet range for a company operating only in the USA, Britain & Europe.

Especially MySQL/MariaDB hide a nasty footgun in this context, as their utf8 charset is a partial UTF-8 implementation using only up to 3 bytes instead of 4. If you want complete UTF-8 support, you instead need to use utf8mb4 as the charset in those databases.

Linux locale settings

Modern linux systems use environment variables to specify what charset is used for text, typically setting the LANG variable to a string containing localization and charset information, like en_US.UTF-8. It can be overridden completely either through the LC_ALL variable, or only individual fields through LC_* variables like LC_CTYPE for the charset. For testing, prefer LC_ALL as it is prioritized over LC_* and LANG.

Any filename or text in linux systems is simply a sequence of bytes (except nullbyte and slash) and the locale decides how to interpret and convert them into printable text.

You can test locale settings on the fly:

touch Café.txt
ls Caf*.txt # prints "Café.txt" on modern linux
LC_ALL=C ls Caf*.txt # always prints "Caf??.txt"
LC_ALL=C.UTF-8 ls Caf*.txt # always prints "Café.txt"

The first ls command uses the default locale setting, UTF-8 for most modern linux distros, the following two set a specific one for consistent output.

Containers often use minimal linux distributions like alpine as the base for their applications to reduce container image size, optimizing storage and bandwidth costs. But minimal distros like alpine do not set a locale by default, so all text is interpreted as ASCII. Remember to set a default locale from the LANG environment variable in these containers. Using LANG=C.UTF-8 is a great starting point to enable UTF-8 support without localizing to any specific region.

Optional UTF-8 BOM errors

The UTF-8 encoding also defines an optional BOM (Byte Order Mark) to identify text as UTF-8, but that notably breaks the backward compatibility with ASCII, so it is rarely ever used.

That said, it is not impossible to encounter in the wild, and can lead to some really annoying errors involving script shebangs.

Executable scripts on linux use a shebang on the first line to declare the program used to execute the rest of the file, looking like this:

#!/bin/bash

This example uses the program /bin/bash to execute the rest of the file contents. The starting sequence !# is important here, as linux expects these exact two bytes followed by the path to the program for the shebang to work.

If the script instead contained an (invisible) UTF-8 BOM before those characters, linux would not recognize the first line as a shebang and instead start executing the script in the default shell, often bash. This might give you cryptic errors that are difficult to debug for unfamiliar developers.

Here is an example: Start by writing a simple python3 script to a file script.py that contains the optional UTF-8 BOM, and make it executable:

printf '\xEF\xBB\xBF#!/usr/bin/python3\nprint("Sample")' > script.py
chmod +x script.py

Printing the file with cat won't show the invisible BOM bytes:

#!/usr/bin/python3

print("Sample")

And inspecting with file -i also seems normal (since it is valid UTF-8):

script.py: text/x-script.python; charset=utf-8

Running it through an interpreter also works:

/usr/bin/python3 script.py

Printing the expected output "Sample".

However, executing the script directly:

./script.py

Will fail with a misleading error message:

./script.py: line 1: #!/usr/bin/python3: No such file or directory
./script.py: line 2: syntax error near unexpected token `"Sample"'
./script.py: line 2: `print("Sample")'

At first glance you may think it didn't find the program /usr/bin/python3, but in reality the file contained a UTF-8 BOM, so the first two bytes were not #! and thus linux did not treat the first line as a shebang at all, instead passing it to the default shell which tried to execute it as shell syntax.

There are two dead giveaways to spot these problems in our example: First, running the file through the python program directly worked, and secondly the shell reports it couldn't find a file starting with #, which should be a comment in shell syntax, so there has to be at least one invisible byte before the # symbol.

Such errors are very difficult to spot and debug in the real world, so knowing of (and being on guard about) the optional UTF-8 BOM in advance may save you some time.

There is no silver bullet

Unfortunately, most of the errors and footguns shown in this article can really only be spotted when you are already familiar with text encodings and their surrounding issues, which typically happens when a production deployment suddenly dies to one of them at 3am. There is no way to avoid them other than already being aware, so having seen the most common ones and faintly remembering that they exist is the best you can do as an engineer today.