Observability & Cost
Every request that reaches Budget AI with a known key is recorded as a trace in the key's project, including requests that were refused. Gateway traces go through the same ingestion path as SDK and OpenTelemetry traces, so they appear in Tracing, Sessions, Users, dashboards and cost reporting with no extra setup.
What a gateway trace contains
| Part | Value |
|---|---|
| Trace id | The gateway request id, returned to the caller in the x-ants-request-id response header |
| Trace name | The model group the request resolved to. When the requested model matched no group, gateway.chat.completions. |
| Tags | gateway, plus any tags the caller sent |
| Generation | One generation named chat.completions (also for embeddings), with model, parameters, input, output, token usage, cost, start time, time to first token and end time |
| Level | DEFAULT when served, WARNING for policy refusals and client disconnects, ERROR for provider and gateway failures |
| User and session | From the caller context headers, see below |
The generation's model is the model the provider reported, falling back to the deployment's provider model. Input is the request's messages (with tools when sent); output is the assistant message, rebuilt from the stream for streamed calls. Both are empty for refusals, errors and usage-only keys.
Recording happens after the response has been sent, on a separate thread in the gateway, so it never slows down or fails a request. One request produces one trace; a client retry is a second trace.
What is not traced
Requests with a missing or unknown key, a malformed or oversized body, or that arrive while the gateway cannot confirm its configuration, have no project to attribute them to and are counted only in gateway metrics. Refusals for blocked, expired or network-confined keys are traced at most once per minute per key and reason.
Finding gateway traffic
- In Tracing, the Source filter offers Gateway and SDK & OTel.
- The
gatewaytag works with the Tags filter. - Opening a gateway trace shows a Gateway panel with key, model group, deployment, provider model, provider, request id and cost, plus the list of attempts and the fallback kind when routing did more than one thing. A usage estimated badge marks traces whose token counts were estimated.
- Custom dashboards can filter and break down by Source, Gateway key and Gateway model group.
Caller context headers
Callers can attribute a request to an end user, a session, tags and their own metadata by sending these optional headers on any request route, including embeddings:
| Header | Format | Becomes |
|---|---|---|
langfuse-user-id | String | The trace's user, shown on the Users page |
langfuse-session-id | String | The trace's session, shown on the Sessions page |
langfuse-tags | Comma-separated list | Added to the trace's tags after gateway |
langfuse-metadata | JSON object of string, number or boolean values | Merged into the trace's metadata |
traceparent | W3C trace context | Recorded as gateway.traceparent metadata only |
If langfuse-user-id is absent, the OpenAI user field from the request body is used as the user.
With the OpenAI Python SDK, pass the headers with extra_headers:
Limits:
| Rule | Limit | When exceeded |
|---|---|---|
User id, session id, traceparent length | 256 characters | That value is dropped |
| Tags | 16 tags of up to 64 characters | Extra or oversized tags are dropped |
| Metadata | A JSON object; 32 keys; values up to 1024 characters | Invalid JSON drops the header; extra keys and long values are dropped |
| Reserved metadata keys | Any key starting with gateway. | Dropped, so callers cannot overwrite gateway fields |
| All caller headers together | 8 KB | All caller context is dropped |
A bad header never fails the request. When anything is dropped, the trace carries gateway.caller_context_rejected naming the rules that tripped. traceparent does not attach the gateway trace to the caller's own trace; it is recorded for reference. Caller context is kept for usage-only keys, because it is attribution rather than model content.
Gateway metadata
Every gateway trace and generation carries these metadata keys (all values are strings):
| Key | Value |
|---|---|
gateway.source | gateway |
gateway.request_id | The request id |
gateway.route | /v1/chat/completions or /v1/embeddings |
gateway.provider | openai-compatible, azure-openai, anthropic, bedrock or vertex-ai; none when refused before a deployment was chosen |
gateway.stream | true or false |
gateway.key_id, gateway.key_alias | The key |
gateway.model_group | The model group requested, or resolved when model was omitted |
gateway.model_group_id | When the group exists |
gateway.deployment_id, gateway.provider_model | The deployment that served, or last failed |
gateway.provider_status | The provider's HTTP status, when it answered |
gateway.refusal | The refusal code, for refused requests |
gateway.attempts | The attempts list, identical to the x-ants-attempts header |
gateway.fallback_kind | general, context or content, when a fallback group served |
gateway.cache | hit for a response cache hit |
gateway.logging_mode | usage_only for usage-only keys |
gateway.usage_estimated | true when the provider reported no usage and tokens were estimated |
gateway.traceparent | The caller's traceparent |
gateway.caller_context_rejected | Caller context rules that dropped a value |
gateway.source_address | The caller's address, on ip_not_allowed refusals only |
gateway.version | The gateway build, when set |
How cost is calculated
- Cost is calculated from the provider's reported token usage and your project's model prices, with the same pricing engine as every other trace. The trace's cost and the key's spend use the same figure.
- Non-streaming responses return the request's cost in the
x-ants-costheader (USD) withx-ants-cost-estimated: false. - Streaming responses send headers before usage is known, so
x-ants-costcarries the estimate made at admission andx-ants-cost-estimated: true. The trace and budgets use the real cost once the stream ends. - When a provider reports no usage, the gateway estimates it (about one token per four bytes) and marks the trace
gateway.usage_estimated. - Embeddings are priced on input tokens.
- Cache hits cost nothing.
- Refusals carry zero usage and are never priced.
- A stream that breaks part-way is still priced for what the provider produced.
A provider model with no price produces traces with no cost, and cannot be used under a hard budget. Add prices for every provider model behind your model groups.
Budget AI analytics
The Budget AI pages read the same trace data, for the time range you select:
| Page | What it shows |
|---|---|
| Overview | Setup status; requests, spend, tokens, error rate, latency p50 with time to first token, refusals, fallbacks and cache hits; traffic and spend over time by model group; refusals by reason; current budgets; Busiest keys; Model health for every deployment; a table of model groups with traffic, latency and deployment health |
| Keys | Key counts (active, blocked, expired, expiring soon, usage only), Traffic by key, Spend by key, and per key: requests, spend against budget, mode, last used and expiry |
| Models & Routing | Traffic by model and Spend by provider model across all groups |
| Model group details | Traffic and latency per model, configured weight against actual share and attempts, keys with access, and errors and refusals grouped by message |
| Budgets & Limits | Spend over time, Budget pressure, budgets with enforced and recorded spend, threshold crossings, and the org unit hierarchy with projections |
| Organization Settings → Budget AI | Requests, cost, error rate and spend against budget for every project, over the last 24 hours |
Overview, key, model and budget figures can lag live traffic by the few seconds it takes traces to be processed. Enforcement does not: limits and budgets are checked against real-time counters.
For wider cost analysis across gateway and non-gateway traffic, use FinOps.