A token budget is a limit on how many tokens an LLM application can consume for a request, conversation, or task—counting prompt tokens, retrieved context, and generated output. It is a primary control for latency, cost, and reliability.
What is Token Budget?
Token usage drives both prefill cost (prompt/context) and decode cost (output). Token budgets set caps or targets for max context and max output, and often include sub-budgets per agent step. In RAG, budgeting forces tradeoffs among top-k retrieval, chunk size, and contextual compression. In agents, budgets prevent infinite loops and tool-call explosions.
Where it’s used and why it matters
Token budgets are used to meet SLAs, cap spend, limit KV-cache growth for concurrency, and enforce product tiers. Effective budgeting pairs with observability so teams can identify wasteful prompts and retrieval configurations.
Examples of Token Budgeting in Practice
- RAG cap: reserve output tokens; allocate the rest to retrieved evidence.
- Adaptive compression: compress retrieved chunks to fit.
- Agent step budget: stop when remaining budget is low.
FAQs
Is this the same as rate limiting? No—rate limiting controls volume over time; token budgets control per-request usage.
How do I choose a budget? Start from cost/latency targets and benchmark real prompts; reserve output headroom.
Biggest failure mode? Truncating critical constraints/evidence; prioritize and summarize instead.
How can I practice? Instrument token counts and experiment with retrieval and compression while measuring success vs. latency.
