Paged attention is an LLM inference technique that manages the transformer’s KV cache using a paging (block-based) memory allocator, similar in spirit to virtual memory paging in operating systems. Instead of storing each request’s key/value tensors in one contiguous GPU memory region, paged attention stores them in fixed-size blocks that can be allocated, reused, and compacted. This reduces memory fragmentation and improves throughput when serving many concurrent, long-context requests.
What is Paged Attention?
In transformer decoding, the KV cache grows with every generated token and must stay in fast memory (often GPU) for attention to reference prior tokens. In a high-concurrency server, requests arrive and complete at different times, producing KV caches of different lengths. If each cache requires contiguous allocation, the GPU memory can fragment: enough total memory exists, but not in a single chunk large enough for a new request.
Paged attention solves this by dividing KV cache storage into uniform blocks (pages). Each sequence’s KV cache is represented as a list of blocks, and the attention kernel reads the logical sequence as if it were contiguous. When a request finishes, its blocks are returned to a free list and can be reused immediately. Because blocks are fixed-size, allocation is fast and predictable, and fragmentation is greatly reduced.
Where it’s used and why it matters
Paged attention is used in LLM serving engines that prioritize high utilization and stable latency under load. It matters most when context windows are long (large KV caches), many sessions run concurrently, and requests have variable lengths and lifetimes.
By improving memory efficiency, paged attention can increase the number of concurrent sequences per GPU and reduce out-of-memory failures. It also enables more practical batching strategies for real-time chat systems, because memory management becomes less of a bottleneck than raw compute.
Examples
- High-concurrency chat: A server handles hundreds of simultaneous conversations; paged attention reduces KV cache fragmentation as users pause and resume.
- Long-context agents: An agent processes long tool traces and documents; paged KV storage helps keep sessions resident without exhausting memory.
- Multi-tenant inference: Different tenants generate different response lengths; block-based allocation avoids worst-case memory waste.
FAQs
Is paged attention the same as KV cache? No. KV cache is the stored key/value tensors; paged attention is a way to allocate and access that cache efficiently.
Does paged attention make generation faster? It mainly improves throughput and stability under load by enabling more concurrent requests; single-request speed improvements depend on the implementation.
When should you use it? When KV cache memory is your primary scaling limit (long contexts, many concurrent users) and fragmentation or allocation overhead is hurting capacity.
