UTF-8 vs UTF-16: Which Encoding Should You Actually Use?

For files, HTTP, JSON, databases, and any general I/O, use UTF-8. Reserve UTF-16 only when a platform or runtime forces your hand, such as native Win32 API calls or Java’s internal string representation. That is the short answer, and the Unicode Consortium and RFC 3629 both point in the same direction.

Why UTF-8 wins by default:

ASCII-compatible. Any valid ASCII file is already valid UTF-8, so legacy toolchains keep working without conversion.
Endianness-neutral. UTF-8 is a byte stream with no byte-order ambiguity. UTF-16 requires a BOM or an out-of-band agreement to avoid misreading bytes.
Self-synchronizing. You can find a character boundary from any byte in a UTF-8 stream. A single corrupted byte in UTF-16 can shift every subsequent character.
Near-universal web adoption. W3Techs surveys consistently show UTF-8 as the dominant encoding across the web.
UTF-16 is required when calling Windows native APIs (which use UTF-16LE internally), working with Java or JavaScript string internals, or reading NTFS filenames through Win32.

UTF-32 exists as a third option: it gives O(1) code-point indexing at a flat 4 bytes per character, but the memory cost rules it out for most production use.

Key Takeaways

UTF-8 is the correct default for virtually all new development; UTF-16 belongs only at platform boundaries where it is explicitly required.

Point	Details
Default to UTF-8	Use UTF-8 for files, HTTP, JSON, and databases; it is ASCII-compatible and endianness-neutral.
UTF-16 is a boundary format	Reserve UTF-16 for Win32 API calls, Java/JavaScript internals, and NTFS filename handling only.
Watch surrogate pairs	Never slice a UTF-16 string at an arbitrary offset; use code-point-aware APIs like `codePointAt()`.
Handle BOMs carefully	Strip BOMs on input; never emit a UTF-8 BOM; always include a BOM or declare endianness for UTF-16.
Validate encoding at every boundary	Declare charset in HTTP headers, set `utf8mb4` in MySQL, and round-trip test any encoding migration.

How does UTF-8 vs UTF-16 actually encode Unicode code points?
When is UTF-8 smaller, and when does UTF-16 win on size?
Compatibility, BOM behavior, and resilience to corruption
Which platforms and languages use UTF-8 vs UTF-16?
Processing costs: indexing, slicing, and memory trade-offs
Common pitfalls: mojibake, invalid sequences, and surrogate bugs
How to choose between UTF-8 and UTF-16 for your project
The case for UTF-8 as the only default worth defending
Sources

How does UTF-8 vs UTF-16 actually encode Unicode code points?

Every Unicode character is assigned a code point, written in U+ notation (for example, U+0041 for the Latin letter “A”). The encoding determines how that abstract number gets stored as actual bytes on disk or in memory.

Code units and byte sizes

UTF-8 uses 8-bit bytes as its code unit. A single code point maps to 1, 2, 3, or 4 bytes depending on its value. UTF-16 uses 16-bit units, so each code point maps to either one unit (2 bytes) or two units (4 bytes). As Wikipedia’s comparison of Unicode encodings notes, UTF-8 uses 1–4 bytes per code point while UTF-16 uses 1–2 sixteen-bit code units (2 or 4 bytes).

The table below maps code-point ranges to their byte costs in each encoding.

Code point range	Plane	UTF-8 bytes	UTF-16 code units (bytes)
U+0000–U+007F	Basic Latin (ASCII)	1	1 unit (2 bytes)
U+0080–U+07FF	Latin extended, Arabic, Hebrew	2	1 unit (2 bytes)
U+0800–U+FFFF	CJK, most BMP characters	3	1 unit (2 bytes)
U+10000–U+10FFFF	Supplementary planes (emoji, rare scripts)	4	2 units (4 bytes, surrogate pair)

Surrogate pairs

Characters above U+FFFF, the supplementary planes, cannot fit in a single 16-bit unit. UTF-16 handles them with a surrogate pair: a high surrogate (U+D800–U+DBFF) followed by a low surrogate (U+DC00–U+DFFF). Together they encode one code point using 4 bytes. RFC 3629 explicitly forbids encoding surrogate code points in UTF-8, which is why you will sometimes see validation errors when surrogate values leak into a UTF-8 stream.

Concrete byte-level examples

Four characters illustrate the difference clearly:

‘A’ (U+0041): UTF-8 → 0x41 (1 byte). UTF-16LE → 0x41 0x00 (2 bytes).
‘é’ (U+00E9): UTF-8 → 0xC3 0xA9 (2 bytes). UTF-16LE → 0xE9 0x00 (2 bytes).
‘世’ (U+4E16, a common CJK character): UTF-8 → 0xE4 0xB8 0x96 (3 bytes). UTF-16LE → 0x16 0x4E (2 bytes).
💩 (U+1F4A9, a supplementary emoji): UTF-8 → 0xF0 0x9F 0x92 0xA9 (4 bytes). UTF-16LE → 0x3D 0xD8 0xA9 0xDC (4 bytes, surrogate pair).

Endianness and the BOM

UTF-16 bytes can be written in little-endian (LE) or big-endian (BE) order. A Byte Order Mark (U+FEFF) at the start of a file signals which order to expect. Windows defaults to UTF-16LE. When no BOM is present, the receiving system must know the byte order through some other agreement, which is a common source of integration bugs.

When is UTF-8 smaller, and when does UTF-16 win on size?

The general rule is straightforward: UTF-8 is smaller whenever the text contains a high proportion of ASCII characters. UTF-16 can be smaller for dense BMP text, particularly CJK scripts, that contains almost no ASCII.

The math in practice

An English paragraph of 1,000 ASCII characters costs 1,000 bytes in UTF-8 and 2,000 bytes in UTF-16. The same 1,000 characters in a CJK-only string cost 3,000 bytes in UTF-8 (3 bytes each) versus 2,000 bytes in UTF-16 (2 bytes each). That is a 33% size advantage for UTF-16 on pure CJK text.

But real documents are rarely pure CJK. HTML markup, JSON keys, CSS, SQL, and code comments are all ASCII. A Chinese web page with standard HTML structure typically contains enough ASCII that UTF-8 comes out comparable or smaller overall, as Wikipedia’s encoding comparison confirms.

Emoji-heavy content is a wash: both encodings use 4 bytes for supplementary-plane characters.

Text profile	UTF-8 size	UTF-16 size	Winner
ASCII-only (English prose)	1 byte/char	2 bytes/char	UTF-8
Mixed ASCII + CJK (typical web page)	~2 bytes/character average	~2 bytes/char avg	UTF-8 or tie
Pure CJK BMP text	3 bytes/char	2 bytes/char	UTF-16
Emoji-heavy (supplementary plane)	4 bytes/char	4 bytes/char	Tie

Edge cases: CESU-8 and modified UTF-8

CESU-8 and modified UTF-8 (used in Java serialization and some legacy systems) encode supplementary characters as two separate three-byte sequences rather than a single four-byte sequence. This inflates size and produces byte sequences that are invalid standard UTF-8. Treat them as internal implementation details, not interchange formats. RFC 3629’s tightened constraints exist partly to prevent these variants from spreading.

UTF-8 wins for any text with significant ASCII content, which covers most web, API, and log data.
UTF-16 has a size edge only for BMP-heavy scripts with minimal ASCII, a narrow real-world scenario.
Supplementary characters cost 4 bytes in both encodings, so emoji-heavy content does not favor either.

Compatibility, BOM behavior, and resilience to corruption

ASCII compatibility

UTF-8’s single biggest practical advantage is backward compatibility. Every valid ASCII file is a valid UTF-8 file, byte for byte. Tools written before Unicode existed, grep, sed, many C libraries, read UTF-8 without modification as long as they only touch ASCII-range bytes. UTF-16 breaks all of them: the null byte in every ASCII character’s high byte confuses C-string functions and the encoding is unrecognizable to tools that expect single-byte text.

BOM behavior

For UTF-16, a BOM is functionally necessary unless endianness is declared out-of-band. Without it, a file labeled only “UTF-16” is ambiguous. For UTF-8, a BOM (the three-byte sequence 0xEF 0xBB 0xBF) is technically legal but widely discouraged. Many tools, including some versions of Python’s CSV reader, older PHP parsers, and various Unix utilities, treat the UTF-8 BOM as literal data rather than a marker, producing a garbage character at the start of the first field. Best practice is to accept and strip BOMs on input but never emit a UTF-8 BOM on output.

Error recovery and self-synchronization

UTF-8 is self-synchronizing. Each byte’s leading bits identify whether it is a single-byte character, a multi-byte lead byte, or a continuation byte. If one byte is corrupted or dropped, the decoder can resync at the next valid lead byte, typically within a few characters. UTF-16 has no such property. Drop a single odd byte and every subsequent character shifts by one byte, producing completely wrong output for the rest of the stream.

Pro Tip: When reading UTF-8 from untrusted sources, use a decoder that replaces invalid byte sequences with U+FFFD (the replacement character) rather than throwing an exception. Python’s errors='replace' or errors='surrogateescape' modes are good examples. For UTF-16, validate that the stream length is even and that a BOM or declared endianness is present before processing.

Practical interoperability checklist:

Always declare charset in HTTP Content-Type headers (charset=utf-8).
Set database column collation explicitly. For MySQL, use utf8mb4, not utf8, which only covers the BMP.
Strip incoming BOMs before processing; never emit a UTF-8 BOM.
Reject or sanitize lone surrogates in UTF-8 streams at the API boundary.
When exchanging UTF-16 files, always include a BOM or document the byte order in the protocol spec.

Which platforms and languages use UTF-8 vs UTF-16?

Platform choice often determines encoding more than personal preference does.

Operating systems

Windows uses UTF-16LE for its internal APIs. The Win32 WCHAR type is a 16-bit unit, and functions like CreateFileW or ReadDirectoryChangesW all operate on UTF-16LE strings. NTFS stores filenames in UTF-16. Linux, macOS, and other Unix-like systems default to UTF-8 for filenames, environment variables, and standard I/O. APFS (Apple’s current filesystem) stores filenames as UTF-8 and normalizes them to NFC form. The older HFS+ used a decomposed UTF-16 variant, which caused notorious filename compatibility issues when transferring files between macOS and Linux.

Programming languages and runtimes

Java and JavaScript use UTF-16 for internal string representation. String.length() in Java and .length in JavaScript return the number of UTF-16 code units, not the number of user-perceived characters. A single emoji can report a length of 2.
Python 3 uses UTF-8 for I/O by default and stores strings internally in a flexible format (Latin-1, UCS-2, or UCS-4) depending on the highest code point in the string.
Go stores strings as raw UTF-8 bytes. The range loop iterates over code points, not bytes.
Rust enforces valid UTF-8 for its str type at compile time. Invalid sequences are a type error, not a runtime surprise.
Swift (since Swift 5) stores String as UTF-8 internally, a deliberate shift from the earlier UTF-16 representation.

Modern language trends are moving toward UTF-8 for both I/O and internal storage, reducing the friction of cross-system text handling.

Web adoption: UTF-8 is the dominant encoding across the web by an overwhelming margin, as character encoding surveys consistently show.

Declaring anything other than UTF-8 for a new web resource requires a specific technical justification.

Windows APIs: UTF-16LE required for native calls.
Linux/macOS filesystems: UTF-8 by default.
Java/JavaScript internals: UTF-16 (watch .length on emoji).
Go, Rust, Swift 5+, Python 3 I/O: UTF-8.
Web and HTTP: UTF-8, universally.

Processing costs: indexing, slicing, and memory trade-offs

Neither UTF-8 nor UTF-16 gives you O(1) code-point indexing. Both are variable-width encodings, which means jumping to the nth character requires scanning from the start or maintaining a cached index. UTF-32 solves this with a fixed 4 bytes per code point, but the memory cost is prohibitive for most workloads, as UTF-8’s Wikipedia entry notes.

Where the costs actually show up

Counting characters, slicing strings, and running regex all require the runtime to walk the byte stream. In UTF-16, BMP characters are one code unit, so BMP-heavy text (most CJK) processes faster per code point than in UTF-8, where those same characters cost 3 bytes. But UTF-8’s smaller footprint for ASCII-heavy text means more of the string fits in CPU cache, which often more than compensates for the extra byte-walking.

The real danger is not speed but correctness. The Unicode FAQ on surrogates documents how JavaScript and Java developers frequently hit bugs where string.length returns 2 for a single emoji, or where substring(0, 5) slices through the middle of a surrogate pair, producing an invalid string. These are silent corruption bugs, not exceptions.

Pro Tip: Use code-point-aware APIs everywhere: String.codePointAt() and String.fromCodePoint() in JavaScript, codePoints() stream in Java, or a grapheme-cluster library like ICU when you need to count what users actually see as characters. Never assume 1 code unit = 1 visible character in any variable-width encoding.

Random access by code point: requires O(n) scan in both UTF-8 and UTF-16.
UTF-16 can be faster per code point for BMP-heavy content (fewer units to walk).
UTF-8 often wins on cache efficiency for mixed or ASCII-heavy content.
Slicing without checking code-unit boundaries corrupts data silently in both encodings.

Common pitfalls: mojibake, invalid sequences, and surrogate bugs

Mojibake

Mojibake is what happens when text encoded in one encoding gets decoded as another. A UTF-8 file opened as Latin-1 produces a string of garbage multi-byte sequences rendered as individual accented characters. The fix is always to declare and enforce encoding at every boundary: file open, HTTP response, database connection, and inter-process pipe.

Invalid UTF-8 sequences

Not every byte sequence is valid UTF-8. Overlong encodings (encoding U+0041 as two bytes instead of one), sequences that exceed U+10FFFF, and lone continuation bytes are all invalid. Different runtimes handle them differently: some throw exceptions, some substitute U+FFFD, and some pass the bytes through silently. Silent pass-through is the dangerous case, because it can allow encoding-based injection attacks where a malformed sequence bypasses a security filter that only checks for the ASCII representation of a dangerous character.

Surrogate-pair splitting

In JavaScript, this code produces a length of 2 for a single emoji:

const s = "💩";
console.log(s.length); // 2, not 1
console.log(s.slice(0, 1)); // broken: half a surrogate pair

The safe alternative uses the iterator protocol or Array.from(s), which respects code-point boundaries. In Java, use codePointCount() instead of length() and codePointAt() instead of charAt() for any string that might contain supplementary characters.

Modified UTF-8, CESU-8, and WTF-8

Modified UTF-8 (used in Java’s DataOutputStream and some JNI interfaces) encodes U+0000 as 0xC0 0x80 rather than 0x00, avoiding null bytes in C-style strings. RFC 3629’s historical notes document CESU-8, which encodes supplementary characters as two three-byte sequences (one for each surrogate half). WTF-8 is an informal name for streams that contain lone surrogates. None of these are valid interchange formats. If you encounter them, convert to standard UTF-8 before processing.

Declare encoding explicitly at every system boundary.
Reject or replace invalid sequences rather than passing them through.
Never slice a UTF-16 string at an arbitrary code-unit offset without checking for surrogate boundaries.
Treat modified UTF-8 and CESU-8 as internal formats only; convert before interchange.

How to choose between UTF-8 and UTF-16 for your project

Work through these questions in order. The first “yes” that applies determines your encoding.

1. Are you calling Win32 or another API that mandates UTF-16?
Use UTF-16LE for that interface. Wrap it with a conversion layer at the boundary so the rest of your application stays in UTF-8.

2. Are you using a runtime whose string type is UTF-16 internally (Java, JavaScript)?
Accept the UTF-16 internals, but convert to UTF-8 for all I/O: file writes, HTTP responses, database inserts, and serialization. Never expose the internal representation as an interchange format.

3. Is the data ASCII-heavy, markup-heavy, or going over a network protocol?
UTF-8. This covers JSON, XML, HTML, CSV, log files, and most REST APIs.

4. Does the database or protocol mandate a charset?
Follow the mandate. For MySQL, specify utf8mb4 to cover the full Unicode range including emoji. For PostgreSQL, UTF8 is the correct collation name. For HTTP, set Content-Type: text/html; charset=utf-8.

5. Do you need constant-time code-point indexing and can afford 4× memory?
Consider UTF-32 for that internal buffer only, and convert at the boundary.

Default recommendations:

Files: UTF-8, no BOM.
HTTP responses: charset=utf-8 in every Content-Type header.
JSON: UTF-8 (the JSON spec mandates it for interchange).
Database columns: utf8mb4 (MySQL) or UTF8 (PostgreSQL).
UTF-16: only at Win32 API boundaries or when a runtime exposes it natively.

Pro Tip: When migrating an existing system from UTF-16 or Latin-1 to UTF-8, write a round-trip validation script before touching production data. Convert a sample, decode it back, and compare byte-for-byte with the original. Include test strings with surrogate pairs, combining characters, and null bytes. A silent mismatch in a filename or database key is far harder to debug after the fact than before.

Document the encoding decision in your project’s README or ADR (Architecture Decision Record) so the next developer does not have to rediscover it.

The case for UTF-8 as the only default worth defending

The conventional framing treats UTF-8 and UTF-16 as roughly equivalent options with different trade-offs. That framing is outdated. UTF-16 made sense in the early 1990s when the Unicode Consortium believed 65,536 code points would be enough for all living scripts, and a fixed-width 16-bit encoding seemed like a clean solution. That assumption collapsed when supplementary planes were added, and UTF-16 became variable-width anyway, inheriting the indexing complexity of UTF-8 without the ASCII compatibility.

What often gets underestimated is how much the Windows UTF-16 legacy shapes developer intuition. Java and JavaScript both adopted UTF-16 internals because they were designed in the mid-1990s Windows ecosystem. That decision has cost the industry decades of surrogate-pair bugs, incorrect .length calculations, and broken emoji handling. The platform shift is already underway: Swift moved to UTF-8 storage in Swift 5, Rust enforces UTF-8 at the type level, and even Microsoft now recommends UTF-8 for new Windows applications where Win32 interop is not required.

The conservative cases for UTF-16 are real and should not be dismissed. If you are writing a Windows shell extension, a JNI bridge, or a Java library that must interoperate with Android’s internal string APIs, UTF-16 is not a choice but a constraint. Handle it at the boundary, convert cleanly, and keep the rest of your system in UTF-8. That boundary-conversion pattern is the right mental model: UTF-16 is an interop format for specific platforms, not a general-purpose encoding for new work.