An LLM router is a routing component that selects which large language model (or which configuration of a model) should handle a given request based on signals such as cost, latency, required quality, safety constraints, language, modality, and domain. Instead of sending every prompt to a single model, the router performs dynamic model selection and dispatch, often with fallbacks and policy checks.
What is LLM Router?
An LLM router sits between an application (or an agent) and one or more model endpoints. It receives the prompt plus metadata (user tier, privacy mode, maximum latency, allowed providers) and then chooses a “best fit” target. Routers can be rule-based (if prompt is short, use a cheaper model), score-based (predict quality or uncertainty), or learned (train a classifier/regressor to pick the model that will satisfy an objective function).
A typical router also manages fallbacks: if the selected model times out, violates a policy, or produces low-confidence output, the router can retry with another model or escalate to a stronger one. Some routers add prompt transformations (system prompts, safety prefixes), enforce structured output requirements, and normalize telemetry for evaluation and cost attribution.
Where LLM Routers are used (and why they matter)
LLM routers are widely used in production GenAI systems to balance quality and cost. For example, a customer-support assistant may route routine FAQ queries to a smaller, cheaper model, while routing complex billing disputes to a more capable model. In agentic workflows, routers help pick specialized models for tool-use, code generation, multilingual responses, or vision tasks.
Routers matter because model choice can dominate user experience and spend. Routing enables predictable latency, cost controls, and reliability via fallbacks, while also simplifying experimentation (A/B tests, canary releases) across providers.
Examples
- Tiered routing: small model for drafts, large model for final answers.
- Safety routing: send sensitive content to a model with stricter safety alignment.
- Domain routing: legal queries to a model fine-tuned on legal text.
- Fallback routing: primary provider → secondary provider on error.
FAQs
Is an LLM router the same as load balancing?
Not exactly. Load balancing spreads traffic across identical resources, while routing selects among different models/configurations based on request intent, constraints, and expected quality.
How does a router decide “quality” ahead of time?
Common methods include prompt classifiers, predicted uncertainty, historical win-rates by category, and online evaluation signals such as user feedback or automated graders.
Do routers increase complexity?
Yes. They add policy, evaluation, and observability requirements, but they often reduce overall cost and improve reliability at scale.
