4 lessons
Continuous Batching
Why batching requests together is essential for throughput, why naive batching wastes it anyway, and how continuous batching fixes that.
PagedAttention and KV Cache Memory Management
Borrowing virtual memory paging from operating systems to stop the KV cache from wasting most of a GPU's memory.
Disaggregated Prefill/Decode Serving
Why the two halves of serving a request fight over the same GPU, and what happens when you give them separate hardware entirely.
Serving Frameworks and Observability
What an inference server actually does end to end, the frameworks that implement it, and the metrics that tell you whether it's healthy.