How LLMs actually read text

A translator who only speaks numbers

Picture a translator who has memorized a giant dictionary, but instead of mapping "cat" to "chat" (English to French), this translator maps every possible word-piece to a unique number. You hand them a sentence, they hand back a list of numbers. That's the very first thing that happens to your text before an LLM ever "thinks" about it.

An LLM, underneath all the intelligence, is a mathematical function: it takes numbers in, and produces numbers out. It has never seen a single letter of the alphabet. Every prompt you write gets converted through a tokenizer into a list of integers, and everything the model outputs is a list of integers converted back into text.

"I love pizza"  ──tokenizer──▶  [40, 3021, 24951]  ──model──▶  [3376]  ──detokenizer──▶  "!"
      (text)                        (token IDs)              (token ID)                (text)

Why not just use one number per letter?

The most obvious idea — assign a number to each of the 26 letters — technically works but is wasteful. "Strawberry" would become 10 separate numbers, and the model would have to learn, from scratch, that the sequence s-t-r-a-w... means a fruit. It would need enormous amounts of data just to relearn spelling.

The opposite extreme — one number per whole word — has the reverse problem: English has hundreds of thousands of words, plus every possible misspelling, name, and made-up word. A fixed vocabulary can never cover everything, and it would explode in size.

The real answer, used by essentially every modern LLM, sits in between: subword tokenization, most commonly a method called Byte Pair Encoding (BPE). Common whole words ("the", "is", "context") get their own single token. Rare or unusual words get split into a few meaningful pieces ("tokenization" might become "token" + "ization"). This gives you a manageable vocabulary size (often 30,000–100,000+ entries) that can still represent literally any string, including typos, code, emoji, and other languages, by falling back to smaller and smaller pieces.

How BPE actually builds its vocabulary

BPE starts absurdly simple and builds itself up through repetition. Here's the whole algorithm:

1. Start by treating every character as its own token.
2. Look at your entire training text and find the pair of adjacent tokens that appears most often.
3. Merge that pair into one new token, add it to the vocabulary.
4. Repeat steps 2–3 thousands of times.

The math: why merge counts matter

At each BPE step, you're computing a frequency count over all adjacent token pairs and picking the maximum:

$\text{pair to merge} = \arg\max_{(a,b)} \; \text{count}(a, b)$

where $\text{count}(a, b)$ is how many times token $a$ is immediately followed by token $b$ across the whole training corpus. You do this over and over, and each merge operation is $O(n)$ over the corpus in a naive implementation, though real tokenizer training uses smarter data structures to avoid rescanning everything every time.

The number of merges you choose to perform directly sets your final vocabulary size:

$\text{vocabulary size} = \text{number of base characters} + \text{number of merges performed}$

A GPT-style tokenizer with roughly 50,000 tokens performed roughly 50,000 minus a few hundred (for the base byte values) merge operations during training.

Hands-on: a minimal BPE trainer in C++

This is a simplified version of the real algorithm — enough to actually see merges happen, without the performance tricks a production tokenizer needs.

#include <iostream>
#include <vector>
#include <string>
#include <map>
#include <sstream>

using TokenSeq = std::vector<std::string>;

// Split a word into a sequence of single-character "tokens" to start.
TokenSeq wordToChars(const std::string& word) {
    TokenSeq result;
    for (char c : word) result.push_back(std::string(1, c));
    return result;
}

// Count how often each adjacent pair of tokens occurs across all words.
std::map<std::pair<std::string, std::string>, int> countPairs(const std::vector<TokenSeq>& corpus) {
    std::map<std::pair<std::string, std::string>, int> pairCounts;
    for (const auto& word : corpus) {
        for (size_t i = 0; i + 1 < word.size(); i++) {
            pairCounts[{word[i], word[i + 1]}]++;
        }
    }
    return pairCounts;
}

// Merge every occurrence of the winning pair into a single new token.
TokenSeq mergePair(const TokenSeq& word, const std::pair<std::string, std::string>& pair) {
    TokenSeq result;
    size_t i = 0;
    while (i < word.size()) {
        if (i + 1 < word.size() && word[i] == pair.first && word[i + 1] == pair.second) {
            result.push_back(pair.first + pair.second);
            i += 2;
        } else {
            result.push_back(word[i]);
            i += 1;
        }
    }
    return result;
}

int main() {
    std::vector<std::string> words = {"low", "lower", "lowest", "low", "low", "lower"};
    std::vector<TokenSeq> corpus;
    for (const auto& w : words) corpus.push_back(wordToChars(w));

    const int NUM_MERGES = 5;
    for (int step = 0; step < NUM_MERGES; step++) {
        auto pairCounts = countPairs(corpus);
        if (pairCounts.empty()) break;

        // Find the most frequent pair.
        auto best = std::max_element(pairCounts.begin(), pairCounts.end(),
            [](const auto& a, const auto& b) { return a.second < b.second; });

        std::cout << "Merge " << step + 1 << ": \"" << best->first.first
                  << "\" + \"" << best->first.second << "\" (count " << best->second << ")\n";

        for (auto& word : corpus) word = mergePair(word, best->first);
    }

    std::cout << "\nFinal tokenization:\n";
    for (size_t i = 0; i < words.size(); i++) {
        std::cout << words[i] << " -> ";
        for (const auto& t : corpus[i]) std::cout << "[" << t << "] ";
        std::cout << "\n";
    }

    return 0;
}

Why this matters for context engineering

Every token costs you space in your context window, and different text "costs" different amounts depending on how it tokenizes:

• English prose: roughly 1 token per 0.75 words on average.
• Code, especially with unusual variable names or heavy indentation: often more tokens per character than prose, because symbols and whitespace tokenize less efficiently.
• Non-English languages, especially ones with fewer training examples: can cost 2–3x more tokens for the same sentence, because the tokenizer's merges were learned mostly from English-heavy text.

This is why a context engineer needs to actually think about token counts, not word counts or character counts, when budgeting what fits.

Common mistakes

1. Assuming 1 word = 1 token. It's a useful rough estimate (used in the last lesson deliberately), but real ratios vary a lot, especially for code and non-English text.
2. Forgetting punctuation and whitespace tokenize too. A sentence with heavy formatting (markdown, JSON) can be surprisingly token-expensive.
3. Assuming every model uses the same tokenizer. GPT models, Claude, and LLaMA all use different vocabularies, so the exact same string can have a different token count on each.

A few more concepts worth knowing

• Special tokens: real tokenizers reserve specific IDs for structural markers like "start of sequence," "end of sequence," or role separators (system/user/assistant) — these aren't ordinary text tokens but are essential plumbing the model relies on to understand structure.
• Vocabulary size trade-off: a larger vocabulary means fewer tokens per sentence (shorter sequences) but a bigger, more expensive embedding lookup table (Module 2) — tokenizer design is itself a balancing act.
• Byte-level fallback: most modern tokenizers (like GPT's tiktoken) operate on raw bytes rather than characters, which guarantees they can represent literally any input, including emoji, unusual Unicode, or corrupted text, by falling back to individual bytes when nothing larger matches.
• SentencePiece and WordPiece: BPE isn't the only subword algorithm in use — SentencePiece (used by LLaMA and many others) and WordPiece (used by BERT) are close cousins with slightly different merge-selection rules, but the same core idea of building a vocabulary from frequent subword units.

Summary

Text never reaches an LLM as text — it's converted into a sequence of integer token IDs by a subword tokenizer, almost always trained with Byte Pair Encoding, which repeatedly merges the most frequent adjacent pair of symbols until it reaches a target vocabulary size. This matters for context engineering because every token — not every word, not every character — eats into your fixed context window budget. Next, we go deeper into that budget itself: exactly what the "context window" is, why it has a hard limit, and where that limit actually comes from.