Prefix caching is an LLM serving optimization that reuses computed model states for a shared prompt prefix—such as a long system prompt, tool schemas, or a fixed instruction block—so repeated requests can skip redundant prompt processing and start generating faster.
What is Prefix Caching?
LLM inference has two phases: prefill (prompt processing) and decoding. If many requests share an identical initial token sequence (the prefix), the server can cache the per-layer key/value tensors produced during prefill for that prefix. When a new request arrives with the same prefix, the server loads the cached states and processes only the request-specific suffix, reducing time-to-first-token and GPU compute.
Prefix caching is especially valuable for long, mostly-static prefixes: tool definitions, policy blocks, formatting schemas, and role instructions. It requires token-level cache keys, cache invalidation when prompts/tools change, and strict tenant isolation.
Where it’s used and why it matters
Prefix caching is used in high-traffic chat and agent runtimes to cut repeated prefill costs. It improves capacity and latency, and complements (but differs from) KV cache: KV cache is per-request during decode, while prefix caching is reused across requests.
Examples of Prefix Caching in Practice
- Tool-heavy agent: cache tool schemas + safety rules.
- Enterprise assistant: cache a static compliance policy prefix.
- Multi-tenant product: cache per-tenant prefixes with isolation.
FAQs
How is prefix caching different from prompt caching? Prefix caching is a common prompt-caching approach focused on a shared prefix; prompt caching may include broader segmentation strategies.
What can break cache reuse? Any tokenization change: model/tokenizer version, whitespace, tool schema order, or prompt edits.
Does prefix caching improve tokens/second? Mostly TTFT; decode speed depends more on attention and KV-cache efficiency.
How do I implement it safely? Version everything, scope caches by tenant, avoid caching private data, and monitor hit rate and latency.
