A shared API makes models more interchangeable. It does not make them identical. Tool calling, structured outputs, caching, safety features, and context management vary between providers. If you ignore these differences, you will only discover the lock-in when you try to switch.
Portability therefore does not mean that another provider will produce identical answers without any changes. It means that switching costs are visible and that the migration can be tested against defined criteria.
Identify every dependency
The model name is only one dependency. Also document:
- prompt and message formats,
- tool and schema capabilities,
- context and caching mechanisms,
- embeddings and vector dimensions,
- moderation and safety filters,
- batch, file, and real-time interfaces,
- logging, regions, and contractual rules.
A migration can depend on a small provider-specific feature even when the main request appears compatible.
Define a common core
Specify which capabilities every production route requires. These may include text output, JSON Schema, tool calling, streaming, or a particular context size.
Provider-specific capabilities should be labelled as deliberate extensions. This makes it clear which routes are readily portable and which require an adapter or redesign.
Version prompts and evaluations
Store prompts outside individual provider consoles and version them together with their associated tests. A provider switch should not be assessed by stylistic similarity, but by whether the workflow fulfils its objective.
An evaluation set should include normal cases, edge cases, and disqualifying errors. Run it against the old and new providers using the same rules.
Data exports and observability
Gateway configurations, logs, cost history, and evaluation results should be exportable. Otherwise, the gateway itself becomes the new dependency.
Use open telemetry formats where appropriate, and maintain your own mapping between workflows, model routes, and result status. Provider dashboards remain useful, but they should not be the only source of information.
Use a parallel route before switching
For important workflows, a temporary shadow route can help. The new provider processes representative tasks without determining the production result. Compare quality, latency, cost, and errors.
Account for data protection and duplicate processing. Shadow operation is only permitted when both data paths have been approved.
Define contractual and technical exit criteria
Specify what must be exported or deleted when the contract ends, how long logs remain available, and what support the provider offers. On the technical side, you need a tested plan for key rotation, DNS or gateway routing, monitoring, and rollback.
Sources
- OpenTelemetry Semantic Conventions for GenAI: https://opentelemetry.io/docs/specs/semconv/gen-ai/
- NIST Cloud Computing Standards Roadmap: https://www.nist.gov/publications/us-government-cloud-computing-technology-roadmap-volume-i-high-priority-requirements
- LiteLLM Provider Documentation: https://docs.litellm.ai/docs/providers
- Cloudflare AI Gateway: https://developers.cloudflare.com/ai-gateway/
Next step
Choose one production workflow and create a capability profile. Test it with a second provider using the same evaluation set. Document every required adjustment and count it as a real switching cost.