The context window, explained properly

A desk, not a filing cabinet

Think of the model's context window as a desk, not a filing cabinet. A filing cabinet can store thousands of documents indefinitely. A desk only has so much surface area — whatever's not physically laid out on it right now might as well not exist while you're working.

The context window is the maximum number of tokens (from the last lesson) a model can have laid out on its desk at once, counting both what you send in and what it's about to generate back. It is not a soft suggestion or a "please keep it short" guideline — it is a hard architectural limit, baked into how the model was built and trained.

Where the limit actually comes from

A transformer-based LLM (we'll build up the actual attention math in Module 3) computes relationships between every pair of tokens in its input. If you have $n$ tokens, the model does work proportional to $n^2$ pairs of tokens comparing themselves to each other:

$\text{attention computation} \propto n^2$

Double the tokens, and the computation roughly quadruples, not doubles. This is why context windows didn't simply start at "infinity" — early transformer models (2017–2019) were commonly limited to 512 or 1024 tokens, because anything larger made both training and inference too slow and memory-hungry on the hardware available at the time.

Modern models have pushed this number dramatically higher (128K, even 1M+ tokens in some 2024–2025 models) through architectural tricks — sparse attention, sliding windows, better position encodings — which we cover properly in Module 6. But every model, no matter how large its window, still has one.

The context window budget equation

Here is the equation that runs, explicitly or implicitly, inside every serious LLM application:

If the sum of everything you want to send, plus room for the model's answer, exceeds $C$, something has to give. The three usual strategies, all of which get entire lessons later in this course, are:

1. Truncate — cut old chat history or less relevant documents.
2. Summarize — compress old information into a shorter form (Module 6).
3. Retrieve selectively — only pull in the most relevant documents instead of everything you have (Module 5).

A worked numeric example

Say a model has a 32,000 token context window. Your system prompt is 300 tokens, you want to reserve 2,000 tokens for the model's response, and your chat history so far is 4,500 tokens. How much room is left for retrieved documents plus the user's new question?

$\text{available} = 32000 - 300 - 2000 - 4500 = 25200 \text{ tokens}$

If a single retrieved document averages 800 tokens, you could fit roughly:

$\left\lfloor \frac{25200}{800} \right\rfloor = 31 \text{ documents}$

...assuming you left literally zero room for the user's own question. In practice you'd reserve some slice of that 25,200 for the question itself and use the rest for retrieval, which is exactly the kind of budgeting decision a context engineer makes explicitly rather than leaving to chance.

Hands-on: a reusable context budget calculator

#include <iostream>
#include <string>

struct ContextBudget {
    int contextWindow;
    int systemPromptTokens;
    int reservedOutputTokens;
    int chatHistoryTokens;

    int available() const {
        int used = systemPromptTokens + reservedOutputTokens + chatHistoryTokens;
        int remaining = contextWindow - used;
        return remaining > 0 ? remaining : 0;
    }

    int maxDocuments(int avgTokensPerDoc, int reserveForQuestion) const {
        int forDocs = available() - reserveForQuestion;
        if (forDocs <= 0 || avgTokensPerDoc <= 0) return 0;
        return forDocs / avgTokensPerDoc;
    }
};

int main() {
    ContextBudget budget{32000, 300, 2000, 4500};

    std::cout << "Tokens available after fixed costs: " << budget.available() << "\n"; 

    int reserveForQuestion = 500;
    int docsThatFit = budget.maxDocuments(800, reserveForQuestion);
    std::cout << "Documents that fit (reserving " << reserveForQuestion
              << " tokens for the question): " << docsThatFit << "\n";

    if (budget.available() == 0) {
        std::cout << "WARNING: no room left. Must truncate or summarize chat history.\n"; 
    }

    return 0;
}

This is a toy version of logic that real frameworks (LangChain, LlamaIndex, and most custom RAG pipelines) run before every single model call — decide the budget first, then decide what content earns a spot inside it.

Context window vs. "max output tokens"

A subtlety worth nailing down: many APIs advertise a context window (say 128K) and separately cap how many tokens the model is allowed to generate in one response (say 4,096). These aren't always the same number, and the output cap is usually much smaller than the full window. When budgeting, you must reserve for the output cap, not assume you can leave just a few tokens of headroom.

Common mistakes

1. Confusing context window with "memory." A 1-million-token context window doesn't mean the model remembers your last conversation — it means it can process up to a million tokens in the current request if you send them.
2. Not reserving output space. Filling the context window entirely with input leaves the model no room to answer, and most APIs will simply error out or truncate the response.
3. Assuming bigger windows remove the need for good context engineering. Even with huge windows, cramming in irrelevant text degrades answer quality (see the "lost in the middle" lesson) and costs real money — most APIs charge per token, input and output both.

A few more concepts worth knowing

• Context window overflow errors: most APIs return an explicit error (rather than silently truncating) when a request exceeds the context window — a context-engineered application should catch this and gracefully trim rather than let the request simply fail.
• Streaming responses: models typically stream output tokens as they're generated (Module 3 covers why decode is sequential) rather than waiting to send the full answer at once — this affects perceived latency (time to first token) even though total generation time is unchanged.
• Needle-in-a-haystack testing: a common benchmark for evaluating a model's real usable context length — hiding a specific fact somewhere in a long document and checking whether the model can retrieve it, directly related to the "lost in the middle" effect covered in Module 4.
• Effective vs. advertised context length: a model might advertise a 1-million-token window but show measurably degraded recall well before that limit — the needle-in-a-haystack test above is exactly how this gap gets measured in practice.

Summary

The context window is a hard limit on how many tokens — input plus reserved output — a model can process in a single call, and it exists because attention computation scales quadratically with sequence length. Every context-engineered application runs some version of a budget equation: window size, minus fixed costs (system prompt, reserved output), equals what's left for chat history, retrieved documents, and the user's actual question. Next, we'll sharpen the distinction between "prompt engineering" (wording a single instruction well) and "context engineering" (architecting the entire information pipeline around that instruction).