What Is Inference Engineering

Two different jobs

Training a model and serving a model are built by different teams, optimized for different goals, and fail in different ways.

Training happens once (or periodically). It runs for days to months on thousands of accelerators. The goal is to reach a target loss as cheaply as possible, and if one run takes an extra six hours, nobody outside the team notices.

Inference happens continuously, forever, for as long as the product is alive. Every single request has a user or another system waiting on the other end. The goal is not "minimize total compute" — it is minimize latency and cost per request while meeting a quality bar, at a request volume that can spike 50x during a product launch.

This is the entire reason "inference engineering" exists as a discipline separate from machine learning research. A researcher who ships a model that scores well on a benchmark has not shipped a product. Someone still has to make that model answer in under a second, on hardware that costs money, at a volume that doesn't bankrupt the company.

Why this is hard, not just an ops task

It's tempting to think of inference as "just deployment" — package the model, put it behind an API, done. Three facts make it much harder than that.

1. Models are enormous relative to hardware memory. A 70-billion-parameter model stored in 16-bit precision needs 140 GB just for weights, before you've processed a single token. A single high-end GPU has 80 GB of memory. The model doesn't fit on one chip — this single fact drives entire subfields (quantization, sharding, offloading).

2. Generation is sequential. Large language models produce output one token at a time, and each new token depends on every token before it. You cannot parallelize across the output the way you can parallelize across the input. This sequential dependency is why decoding is often memory-bandwidth-bound rather than compute-bound — a fact that shapes nearly every optimization in this course.

3. Traffic is unpredictable and requests are heterogeneous. One request asks for a 5-token classification label; another asks for a 4,000-token essay. Batching them naively wastes either latency or throughput. A production system must handle both without operators manually tuning batch sizes per request type.

The vocabulary

A handful of terms recur throughout this course.

Latency — time from request to response. Split into two numbers that behave completely differently:

• Time to first token (TTFT) — how long before the user sees anything. Dominated by the prompt processing pass (prefill).
• Time per output token (TPOT), also called inter-token latency — how long between each subsequent token. Dominated by memory bandwidth during decode.

Throughput — total tokens generated per second across all concurrent requests, on a given piece of hardware. This is what determines cost per token.

Prefill vs. decode — the two phases of serving one request. Prefill processes the entire input prompt in one parallel forward pass (compute-bound). Decode generates output tokens one at a time in a loop (memory-bandwidth-bound). Nearly every serving optimization treats these two phases differently because their bottlenecks are opposite.

Batch size — how many requests are processed together in one forward pass. Larger batches amortize the cost of loading model weights across more requests, raising throughput — up to the point where the GPU runs out of memory or compute saturates.

Quantization — representing model weights (and sometimes activations) with fewer bits than they were trained in, trading a small amount of accuracy for large reductions in memory and bandwidth use.

KV cache — memory that stores the intermediate attention states of every previous token in a sequence, so the model doesn't recompute them at every decoding step. Covered in depth in the next lesson.

The two bottlenecks: compute-bound vs. memory-bound

This distinction underlies almost every decision in this course, so it's worth internalizing early.

A GPU has two relevant numbers:

• FLOPS — how many floating-point operations it can do per second (compute).
• Memory bandwidth — how many bytes it can move from memory to its compute units per second.

An operation is compute-bound if the GPU spends more time doing math than waiting for data. It's memory-bound if the GPU sits idle waiting for weights or activations to arrive from memory, even though it has spare FLOPS.

The arithmetic intensity of an operation — FLOPs performed per byte moved — tells you which regime you're in. Prefill has high arithmetic intensity (many tokens reuse the same loaded weights in one pass), so it's compute-bound. Single-token decode has very low arithmetic intensity (one token's worth of math per full weight load), so it's memory-bound.

This is why doubling GPU compute barely speeds up decoding, but doubling memory bandwidth (or shrinking the model with quantization) speeds it up almost linearly. It's also why batching decode requests together helps so much: batching increases the amount of math done per weight load, pushing decode back toward the compute-bound regime.

A mental model to carry forward

Think of serving a large model like running a factory with one enormous, expensive machine (the accelerator) that:

• Takes non-trivial time to load a new "recipe" (the model weights) into its working area
• Can process many small orders at once nearly as fast as processing one, if they use the same recipe
• Produces output one unit at a time when the units depend on each other

Every technique in this course is either about shrinking the recipe (quantization, distillation), keeping more useful state around so it isn't recomputed (KV caching), grouping orders cleverly (batching, scheduling), or splitting the recipe or the work across more machines (sharding, disaggregation). Once you can place a new idea into one of those four buckets, the whole field becomes much less overwhelming.

What this course covers

• Fundamentals (this module) — the mental model, the autoregressive decoding loop, and the KV cache.
• Model Optimization — quantization, distillation, pruning and speculative decoding: techniques that make the model itself cheaper to run.
• Serving Systems — continuous batching, PagedAttention, and the architecture of a modern inference server.
• Hardware & Compilation — what's actually happening on the GPU, from tensor cores to kernel fusion and FlashAttention.
• Advanced Serving Techniques — multi-LoRA serving, guided decoding, request routing and semantic caching, and test-time compute.
• Benchmarking & Evaluation — how to measure whether any of this actually worked, with the exact metrics and formulas serving teams use.
• Scaling & Deployment — multi-GPU parallelism, autoscaling, and choosing hardware and serving frameworks for a real workload.

Summary

Inference engineering is the discipline of making a trained model answer quickly and cheaply at scale, and it is a genuinely different problem from training. The core tension is that models are too large for the memory and bandwidth available, and generation is inherently sequential. Nearly every technique you'll meet in this course reduces to shrinking what has to move through memory, reusing computation instead of repeating it, or spreading work across more hardware.