Mã Hóa Ký Tự UTF-8, UTF-16 Và UTF-32: Bản Chất Byte & Tránh Lỗi Font

Text encoding bugs—mojibake characters like é instead of é—happen when software misinterprets the byte sequences representing Unicode code points. Understanding how UTF-8, UTF-16, and UTF-32 encode characters is essential for building resilient file processing pipelines.

The Unicode Model: Code Points vs Bytes

Unicode maps abstract characters to numeric code points (e.g. U+0041 for 'A', U+1F600 for '😀'). The transformation format (UTF) dictates how those numbers are serialized into binary bytes.

How UTF-8 Variable Length Works

• ASCII characters (0-127): Encoded in 1 byte (0xxxxxxx), maintaining 100% backwards compatibility with legacy ASCII.
• Latin, Greek, Cyrillic (128-2047): Encoded in 2 bytes.
• Asian scripts, CJK (2048-65535): Encoded in 3 bytes.
• Emojis & rare symbols (65536+): Encoded in 4 bytes.

// Decoding bytes safely in Node.js
import { TextDecoder } from 'util';

export function decodeTextSafely(buffer: Buffer): string {
  // fatal: true throws an exception instead of inserting  replacement characters
  const decoder = new TextDecoder('utf-8', { fatal: true });
  return decoder.decode(buffer);
}

The Byte Order Mark (BOM)

UTF-8 does not require a Byte Order Mark because bytes are ordered identically on big-endian and little-endian machines. However, some Windows software (Notepad, Excel) prepends 0xEF 0xBB 0xBF. Always strip leading BOM bytes when parsing text files in Node.js.