Why context windows cost quadratically (and what the KV cache does about it)

The precise reason doubling your prompt length more than doubles the work, worked out from the attention matrix itself, plus the caching trick that makes generation bearable.