A token dashboard shows how much a model consumed. It does not show whether the workflow succeeded. A low-cost model call can be part of an expensive process if retries, tool failures, or human rework follow.
AI observability must therefore connect model telemetry with business outcomes.
The shared trace ID
Every workflow needs an ID that persists across the gateway, retrieval, model, tools, and target system. Without this link, costs can only be analysed by provider or API key.
The trace ID should also be passed to background jobs and agent handoffs. A new process must not appear to be an independent workflow when it belongs to the same customer task.
Which signals should you capture?
For a model call, the relevant signals are model version, provider, tokens, cache status, latency, errors, and cost. For the overall workflow, also capture:
- retrieval sources and number of results,
- tool calls and external API costs,
- retries and fallbacks,
- human approvals and corrections,
- stop or success status,
- domain-specific quality assessment.
Store sensitive content only when it is necessary for the purpose. Metadata and references are often sufficient.
Define success in business terms
"Request completed" is not a business outcome. Define a status for each workflow that reflects the task: ticket classified correctly, document extracted completely, response approved, or error resolved.
Only then can you calculate cost per successful task. Failed attempts must be included in this metric rather than disappearing from the analysis.
Bring quality and cost together
A model can be less expensive and still create more rework. Add evaluation results, reasons for rejection, and correction time. The relevant question is: what does a result cost when it meets the agreed quality threshold?
This perspective prevents teams from celebrating local optimisations while the overall process becomes more expensive.
Baselines and changes
First establish a baseline for each workflow. Then change routing, the prompt, cache, or retrieval one at a time and compare against the same test cases and production signals.
On 29 July 2026, OpenAI reported improvements of 20 per cent in its own serving costs and more than 15 per cent in token generation efficiency. Such provider improvements are relevant, but they do not replace measurement within your own workflow. Lower serving costs do not necessarily translate proportionally into customer prices or process costs.
Alerts that trigger action
Warnings should be tied to specific deviations: rising cost per successful outcome, unusual retry rates, increased fallback usage, declining cache hits, or more frequent human rejections. An alert without an accountable person and a defined response path merely adds more dashboard noise.
Sources
- OpenTelemetry Semantic Conventions for GenAI: https://opentelemetry.io/docs/specs/semconv/gen-ai/
- FinOps Foundation, FinOps for AI: https://www.finops.org/wg/finops-for-ai/
- Cloudflare AI Gateway Analytics: https://developers.cloudflare.com/ai-gateway/observability/analytics/
- OpenAI on X, 29 July 2026: https://x.com/OpenAI/status/2082577277246972300
Next step
Select one production AI workflow and define an unambiguous success status. Connect model, retrieval, and tool costs through a trace ID. Capture a two-week baseline before changing routing or models.