UTF-8 and encoding problems: why accents break and how to fix them
What UTF-8 is, code points, mojibake, and why accented text turns to garbage — the mechanism explained end to end.
You open a text file, an API response, a database field — and where "configuration" should be, you get configuração. Or worse: a lonely question mark where every accented character used to be. The file isn't corrupted. The data isn't gone. Somewhere along the way, someone interpreted the bytes with the wrong encoding.
This kind of failure is silent in the worst sense: the system accepts it, stores it, returns it — and only when a human looks at the screen does something seem off. Understanding the mechanism takes about twenty minutes. Debugging without understanding can take weeks.
What UTF-8 actually is
Before UTF-8, there was a babel of regional encodings. ASCII covered English without accents (128 characters, 7 bits). Latin-1 (ISO-8859-1) extended to 256 and covered Portuguese, Spanish, French. Windows-1252 was a proprietary Latin-1 variant. Shift-JIS covered Japanese. Big5 covered traditional Chinese. None of them talked to each other.
Unicode emerged as the attempt to have a single address space for all characters across all writing systems in the world. Each character gets a number — called a code point — in the format U+XXXX. The letter ç is U+00E7. The emoji 🔥 is U+1F525. The yen symbol ¥ is U+00A5.
Unicode defines the code points. UTF-8 is how those numbers get encoded into bytes.
How UTF-8 maps code points to bytes
UTF-8 is a variable-width encoding. A character takes between 1 and 4 bytes depending on its code point:
| Code point range | Bytes | Example |
|---|---|---|
| U+0000 to U+007F | 1 byte | A (U+0041) → 0x41 |
| U+0080 to U+07FF | 2 bytes | ç (U+00E7) → 0xC3 0xA7 |
| U+0800 to U+FFFF | 3 bytes | € (U+20AC) → 0xE2 0x82 0xAC |
| U+10000 to U+10FFFF | 4 bytes | 🔥 (U+1F525) → 0xF0 0x9F 0x94 0xA5 |
The key detail: the first 128 characters — all of ASCII — are identical in UTF-8. Plain English files that worked in ASCII still work in UTF-8 unchanged. This was an intentional backwards-compatibility decision.
The practical consequence: ç in UTF-8 takes two bytes (0xC3 0xA7). In Latin-1, it takes one byte (0xE7). When you mix the two, the bytes are the same but the interpretation differs — and that's where the trouble starts.
Mojibake: the name for encoding garbage
Mojibake (文字化け) is the Japanese term for the phenomenon of text turning into garbage due to encoding mismatch. You probably recognize its classic forms:
São Pauloread as Latin-1 when it's in UTF-8 →São Pauloconfigurationread as UTF-8 when it's in Latin-1 →configuração- Accented text decoded as ASCII (which doesn't support accents) →
descri??o
The logic is always the same: bytes were written with one encoding and read with another.
# UTF-8 read as Latin-1 (classic mojibake)
utf8_text = "configuration".encode("utf-8")
# plain ASCII, no change here — let's use accented text
utf8_text = "configuração".encode("utf-8")
# b'\x63\x6f\x6e\x66\x69\x67\x75\x72\x61\xc3\xa7\xc3\xa3\x6f'
mojibake = utf8_text.decode("latin-1")
# 'configuração'
# Reversing it: re-encode as latin-1, decode as utf-8
recovered = mojibake.encode("latin-1").decode("utf-8")
# 'configuração'
This recovery trick works only when the original conversion was Latin-1 ↔ UTF-8 and the original bytes weren't altered. If there was truncation, invalid character substitution, or more than one bad conversion in the chain, the data may be unrecoverable.
Why accents break: three failure points
The path of an accented text involves at least three encoding contracts: where it's written, where it's stored, and where it's read. If any point diverges from the others, you have a problem.
1. Database with the wrong charset
MySQL has a historical quirk that has destroyed data for many people: MySQL's utf8 collation is not real UTF-8 — it only supports up to 3 bytes per character, which excludes emojis and some rare characters above U+FFFF. The correct UTF-8 in MySQL is called utf8mb4.
-- Check table charset
SHOW CREATE TABLE users;
-- Fix table and columns
ALTER TABLE users
CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
-- Make sure the connection is also utf8mb4
SET NAMES utf8mb4;
If a table is in latin1 and you insert UTF-8 text, MySQL stores the bytes literally without converting — you'll read them back correctly as long as the connection is also Latin-1, but the moment you switch the connection to UTF-8 you'll see mojibake in everything with accents.
2. File saved in the wrong encoding
The text editor, the Python script, the CSV generator — all need to write the file in the correct encoding. On Windows, the historical trap is that Notepad and many editors saved as Windows-1252 by default. Today, VSCode saves in UTF-8 by default, but Excel still generates CSV in cp1252 when you "Save As" on a Windows system in a non-English locale.
To check a file's encoding without opening it in an editor:
# Linux/macOS
file -i my-file.csv
# my-file.csv: text/plain; charset=iso-8859-1
# Python
import chardet
with open("my-file.csv", "rb") as f:
result = chardet.detect(f.read())
print(result)
# {'encoding': 'ISO-8859-1', 'confidence': 0.73, 'language': ''}
Note the 73% confidence: automatic encoding detection is probabilistic. For ambiguous files (with few special characters), it can be wrong. If the file comes with documentation about its encoding, use that — don't rely solely on auto-detection.
3. Missing HTTP header or meta tag
For web content, encoding needs to be declared explicitly. A server returning HTML without Content-Type: text/html; charset=utf-8 is delegating to the browser the task of guessing. Most modern browsers default to UTF-8, but older versions and some scraping parsers don't.
<!-- In the HTML <head> -->
<meta charset="UTF-8">
# In a Flask API
return Response(json.dumps(data, ensure_ascii=False),
content_type='application/json; charset=utf-8')
The ensure_ascii=False in Python is critical: by default, json.dumps escapes all non-ASCII characters as \uXXXX. The resulting JSON is technically valid but unreadable for debugging and unnecessarily larger.
Guaranteeing UTF-8 end to end
The simplest strategy: don't mix encodings. Set UTF-8 as the default at every layer and never change it.
Practical checklist:
- Database:
utf8mb4charset,utf8mb4_unicode_cicollation (MySQL/MariaDB) orUTF8default (PostgreSQL — already UTF-8 by default on creation) - Database connection:
SET NAMES utf8mb4(MySQL) orclient_encoding = UTF8(PostgreSQL) - Source files: UTF-8 without BOM in editor (check editor settings)
- Data files (CSV, TXT): UTF-8 with or without BOM depending on the consumer (Excel on Windows needs BOM to recognize automatically)
- APIs:
Content-Typeheader with explicitcharset=utf-8 - HTML:
<meta charset="UTF-8">as the first tag in<head> - Python:
open(file, encoding="utf-8")— never trust the system's default encoding
The BOM (Byte Order Mark) issue deserves attention: BOM in UTF-8 (0xEF 0xBB 0xBF) is optional and technically unnecessary — UTF-8 has no endianness concern like UTF-16 does. But some consumers, especially Excel on Windows, use the BOM to automatically detect UTF-8. If you control generation and know the destination is Excel, adding BOM is pragmatic. If the destination is anything else, BOM can cause problems (parsers that don't expect three extra bytes at the start).
The connection to CSV files
CSV deserves special mention because it concentrates all these problems in one place: no metadata header declaring the encoding, generated by disparate tools (Python, Excel, legacy systems), and circulated between teams that never discussed encoding.
If you're dealing specifically with CSV breaking in a data pipeline, the post Importing CSV with wrong encoding: how to diagnose and fix covers the practical cases with Excel, Python, and automatic detection in detail. What I describe here is the underlying mechanism — which explains why those solutions work.
Frequently asked questions
Why does ç specifically become ç?
Because ç in UTF-8 is 0xC3 0xA7. When those two bytes are interpreted as Latin-1: 0xC3 is à and 0xA7 is §. It's a deterministic byte-by-byte translation — every time you see ç, it's ç in UTF-8 being read as Latin-1. Similarly, ã (U+00E3) in UTF-8 is 0xC3 0xA3, which in Latin-1 is ã. That's why ão becomes ão — it's the most recognizable mojibake pattern in Portuguese.
How do I programmatically detect if a string has mojibake?
There's no 100% reliable way, but some heuristics help: presence of sequences like ç, ã, é, á, Ã, â indicates UTF-8 read as Latin-1. The opposite — individual characters outside ASCII where the file should be Latin-1 — is harder to detect automatically. For production data, the best approach is to establish the encoding convention from the start and never depend on detection.
What's the real difference between utf8 and utf8mb4 in MySQL?
utf8 in MySQL supports code points up to U+FFFF (3 bytes per character). utf8mb4 supports up to U+10FFFF (4 bytes), covering emojis and rare scripts. The practical difference: if a user inserts an emoji into a TEXT field with utf8 charset, MySQL will either reject with an error or silently truncate depending on sql_mode. Always use utf8mb4 in new projects — the storage overhead is negligible and you avoid surprises.
Does Python 3 handle this automatically?
Partially. Python 3 treats internal strings as Unicode (UTF-32 in memory), which eliminates the problem inside the process. The problem reappears at boundaries: when reading a file, open() uses the OS default encoding if you don't specify it (locale.getpreferredencoding()). On Windows, that might be cp1252. On Linux with a UTF-8 locale, it's UTF-8. Always specify encoding="utf-8" explicitly when opening files for consistent behavior across environments.
Bytes don't lie — the interpretation does
The data doesn't break on its own. When you see mojibake, the bytes are there, intact. What failed was the interpretation contract at some point in the chain — an unspecified setting, a tool with a different default, a database field with the wrong charset.
The fix isn't magic: it's setting UTF-8 as the explicit standard at every layer instead of relying on defaults. To inspect code points and see how specific characters translate to bytes, the ASCII Table Generator shows the mapping for each character with its decimal, hexadecimal, and binary values — useful when you're trying to understand why a specific byte is producing the wrong character.
The right time to think about this is before the first deploy, not after the first user complaint about broken accents in production.
- 01 Does the PageSpeed Score Affect SEO? 100/100 and Failing Google doesn't rank on PageSpeed's 0 to 100 score but on Core Web Vitals from real users. Here's why a site scores 100/100 and still fails in the field.
- 02 Unit test vs integration test: the difference in practice The difference between unit and integration tests with API and database examples: what each one catches, what it costs, and when to use each in your project.