Post

How Computers Handle Text: A Journey

Posted
By Mahrez Benhamad
3 min read
How Computers Handle Text: A Journey

I think all of us have run into those annoying ???? characters at some point. It’s frustrating, right? But those ???? aren’t just random errors—they’re a window into the fascinating history of how computers handle text. Let’s take a step back and trace the journey, starting from the earliest days when “text” wasn’t even a concept, all the way to the Unicode-powered world we rely on today.

1. The Dawn of Computing: Punch Cards and Binary

In the early days, there was no “text” as we understand it. Computers operated purely in binary, and any semblance of text was represented through hardware-specific conventions like punch cards.

  • Punch Cards: Rectangular pieces of stiff paper where each column could represent a bit (1 for a hole, 0 for no hole).
    • A typical punch card had 80 columns. Patterns of holes encoded data, instructions, or, in some cases, text.

Example:
On one machine, a specific hole pattern might mean A, but on another, it could mean Z. Text was purely a matter of interpretation.

  • The Problem: Every machine had its own system. There was no standard, and transferring data between systems was nearly impossible.

2. The Birth of Standards: ASCII and 7-Bit Encodings

By the 1960s, the world needed a common language for text, and ASCII (American Standard Code for Information Interchange) emerged as the solution.

  • Key Features:
    • 7 bits per character (128 possible symbols).
    • Included:
      • Uppercase and lowercase letters (A-Z, a-z).
      • Numbers (0-9).
      • Basic symbols (e.g., @, &, $).
      • Control codes (e.g., CR for carriage return, LF for line feed).
  • How It Worked:
    Characters were represented by numbers in memory:
    • A → 65 (decimal) → 01000001 (binary).
    • a → 97 (decimal) → 01100001 (binary).

3. The Limitations of ASCII: Where’s the É, Ü, or 日?

ASCII worked for English but ignored most other languages. Developers needed more characters to represent accented letters, non-Latin scripts, and symbols. ASCII couldn’t handle them, and regional encodings emerged.

4. The Expansion: 8-Bit Encodings and ISO-8859

By the 1980s, systems moved to 8 bits per character, doubling the possible symbols to 256. This gave rise to extended ASCII encodings, like ISO-8859.

  • ISO-8859-1 (Latin-1):
    • Added characters for Western European languages (é, ü, ç, etc.).
    • Reserved the top 128 values (128–255) for these extra characters.

Example:

  • é → 233 (decimal) → 11101001 (binary).

5. The Chaos of Encoding Mismatches

Every region had its own 8-bit encoding:

  • Western Europe used ISO-8859-1.
  • Russia used KOI8-R.
  • Japan had its own multibyte systems (e.g., Shift JIS).

The result? Encoding mismatches:

Example:

  • Save a file in ISO-8859-1 with Héllo!.
  • Open it in KOI8-R:
    • Expected: Héllo!.
    • Actual: Hйllo!.

6. The Game-Changer: Unicode

By the late 1980s, it was clear that the world needed a universal system. Unicode was introduced, aiming to give every character from every language a unique code point.

  • How Unicode Works:
    • A → U+0041.
    • é → U+00E9.
    • 日 → U+65E5.

Unicode itself is just a map of code points. To store or transmit text, you still need an encoding format, which led to UTF-8.

7. The Savior of Encodings: UTF-8

Developed in 1993, UTF-8 revolutionized text handling:

  • Backward-Compatible: ASCII characters (0–127) use a single byte, just like in ASCII.
  • Variable-Length Encoding: Characters can use 1 to 4 bytes:
    • A → 41 (1 byte).
    • é → C3 A9 (2 bytes).
    • 😊 → F0 9F 98 8A (4 bytes).

Why UTF-8 Won:
It supported every character in Unicode, minimized storage for common characters, and was compatible with existing ASCII-based systems.

8. The Legacy of ??????: Where Does It Come From?

The infamous ???? appears when:

  1. A Unicode-encoded file (e.g., UTF-8) is read with the wrong encoding (e.g., ISO-8859-1).
  2. Bytes that don’t match the expected encoding are replaced with replacement characters (? or �).

Example:

  • UTF-8 bytes for 😊 (F0 9F 98 8A) are invalid in ISO-8859-1.
  • The result? ??????.

Summary

From punch cards to UTF-8, the history of text encoding reveals humanity’s quest for seamless communication. It’s not just about fixing garbled text—it’s about understanding how computers interpret and handle the written word. Mastering this history ensures your software communicates flawlessly across languages and platforms. Respect it, and your code will never get lost in translation.

development
development backend
This post is licensed under CC BY 4.0 by the author.