Model routing is a policy for choosing the cheapest and fastest execution path that still clears a task-specific quality gate. It is not “send short prompts to a small model.” Prompt length is a poor proxy for difficulty, and a router can save money while concentrating failures in rare or high-risk cases.
Start with a fixed baseline, label representative tasks, and introduce routing only when you can compare quality, latency, and cost per successful task across every route.
Separate routing from cascading
- Routing chooses a model before generation using task metadata or a classifier.
- Cascading runs a cheaper model first and escalates when its result fails a verification gate.
- Fallback changes providers or models after availability failure; it is not a quality policy.
Google's architecture guidance recommends model routing based on task complexity, cost, or latency, while noting that larger thinking budgets can increase both latency and cost. Its gateway documentation also identifies constraints such as common host requirements and cold starts. See Google Cloud's component guidance and model-routing overview.
Write the routing table first
| Task slice | Default route | Escalation gate | Hard constraint |
|---|---|---|---|
| Deterministic extraction | Small model with strict schema | Schema or semantic validation fails | No unsupported field passes |
| Low-risk classification | Small model | Margin below calibrated threshold | Track minority-class recall |
| Grounded answer | Mid-tier model plus retrieval | Citation or groundedness fails | Only allowed sources enter context |
| Consequential recommendation | Strong model plus human review | Always reviewed | Model cannot execute action |
| Complex synthesis | Strong model | Budget or evidence insufficiency | Report missing evidence |
The table makes business risk part of routing. A cheap route may be fine for reversible formatting and unacceptable for an account-closure recommendation.
Optimize cost per successful task
effective_cost(route) = request_cost(route) / task_success_rate(route)cascade_cost = first_pass_cost+ escalation_rate × escalation_cost+ review_cost
A route costing $0.01 with 50% task success has an effective cost of $0.02 before retries and review. Use current provider prices and measured token distributions.
The FrugalGPT paper formalized LLM cascades as a cost-performance tradeoff across model APIs. Its benchmark results apply to that setting, not automatically to yours. See FrugalGPT.
Route on observable features
Prefer features available before generation: authenticated product surface, task type, language, modality, required tools, context size, consequence class, and historical difficulty for the same labeled slice.
Do not route on protected attributes or opaque user-value scores. Do not let user text directly select a privileged model or bypass a policy route.
A model-based router adds latency, cost, and failure. Compare it with deterministic rules. If five product surfaces map cleanly to five task types, a classifier may add no value.
Calibrate escalation on held-out cases
Use deterministic validation, retrieval sufficiency, calibrated confidence, or a task-specific grader. Self-reported model confidence is not enough.
| Metric | Why it matters |
|---|---|
| Task success by slice and route | Detects harm concentrated in a minority slice |
| Escalation precision | Measures unnecessary expensive escalation |
| Escalation recall | Measures first-pass failures that were caught |
| Cost per successful task | Connects spend to outcomes |
| P50/P95 latency | Cascades can hurt tail latency |
| Route stability | Detects classifier or distribution drift |
Store route policy version, selected model, reason code, gate result, escalation, and terminal outcome on one trace. The LLM observability guide provides that structure.
Roll out with shadow decisions
- Run the current fixed model as control.
- Compute the proposed route without applying it.
- Compare decisions with labeled outcomes and costs.
- Enable the cheap route for low-risk slices behind a flag.
- Keep a control sample on the baseline model.
- Alert on quality, escalation recall, and tail latency by slice.
Routing is justified when the policy remains simpler than the savings it creates. If traffic is low or tasks are homogeneous, pick one evaluated model. For broader cost work, use the LLM cost optimization playbook.
