Using a frontier model as the default seems simple. It avoids early routing decisions and often produces good results in demos. In production, however, this choice means that classification, extraction, research, and complex decisions all use the same cost and latency tier.
Good routing does not start with a price list. It starts with clearly separated task types and a measurable quality threshold.
Route tasks, not departments
"Marketing" and "Support" are not model routes. A single department carries out very different tasks: detecting language, extracting fields, summarising documents, drafting responses, or explaining a sensitive recommendation.
Define routes based on the specific work. The narrower the task type, the more accurately you can measure quality, cost, and errors.
Four criteria for a route
A route should consider at least:
- difficulty and the required depth of reasoning,
- the impact of an error,
- data class and permitted providers,
- latency and cost budget.
Context length, tool calling, and structured outputs add technical requirements. A low-cost model is not an option if it cannot reliably meet the required output format.
Evaluate first, then route
Create a small, realistic evaluation set for each task type. It should include normal cases, difficult cases, and clearly defined disqualifying errors. Compare models using the same inputs and assessment criteria.
Route by cost only among models that meet an acceptable quality threshold. If a model causes more retries or human corrections, it may be more expensive despite lower token prices.
Static and dynamic selection
Static routing assigns a task type to a fixed model. It is easy to review and is the best starting point for many enterprise workflows.
Dynamic routing estimates difficulty or risk for each request. It can be more efficient, but it is itself a model or rule-based system that must be evaluated. In particular, test whether sensitive or difficult cases are reliably escalated.
Escalation instead of invisible repetition
When a low-cost model fails, the workflow should not switch indefinitely to more powerful models. Define an escalation chain with a maximum number of attempts and visible cost attribution.
For critical tasks, the final step may be human review. The gateway log should show why an escalation occurred and which model produced the final result.
Model updates are production changes
Providers change models, versions, and default behaviour. A new version must not automatically take over every route. Run the evaluation set again, compare quality, latency, and cost, and document the decision.
The same applies to local or open models. A new benchmark result is no substitute for testing on your own tasks.
Sources
- OpenAI Evals: https://platform.openai.com/docs/guides/evals
- Anthropic, Define success criteria and build evaluations: https://docs.anthropic.com/en/docs/test-and-evaluate/define-success
- LiteLLM, Routing and load balancing: https://docs.litellm.ai/docs/proxy/load_balancing
- NIST AI Risk Management Framework: https://www.nist.gov/itl/ai-risk-management-framework
Next step
Select three frequent task types with different risk levels. Create ten to twenty real test cases for each, define disqualifying errors, and compare two model tiers. Only then should you route based on cost per accepted outcome.