Model Groups & Routing

Applications never name a provider model directly. They name a model group, and Budget AI decides which provider model serves each request. You can change providers, add capacity or fail over without touching application code.

Concepts

The same provider model can appear more than once in a group with different credentials. That is how you spread traffic or fail over between two accounts or two regions.

Create and manage model groups

All of this is in Budget AI → Models & Routing → Model groups and needs gatewayAi.models:admin.

On each model you can Edit the credential, provider model or weight, Pause or Resume it, or remove it. Pause stops traffic to a model under every strategy. A group can be deleted only when it has no models and no keys are granted it.

Select Details on a group for its own page: traffic and latency per model, configured weight against actual share, keys with access, errors and refusals grouped by message, and the routing policy.

Changes reach every gateway replica within about a second. See configuration propagation.

Routing strategies

The strategy decides which eligible model serves a request. Paused models are never eligible. Benched models are skipped (see Benching and cooling-off).

Latency and load are measured on each gateway replica. Under the three non-weighted strategies, a model with fewer than 20 measured requests receives an exploration share of about 10 % of traffic until it has a history, weight only breaks ties, and a weight of 0 does not stop traffic. Use Pause for that.

If no model is eligible because every model is paused, or every model has weight 0 under Weighted, the request is refused with 503 no_deployment_available.

Retries, timeouts and failover

Each group has a routing policy:

When a call fails, Budget AI escalates in this order:

1. Retry the same model while the failure is retryable and retries remain.
2. Try the next eligible model in the group.
3. Try the group's fallback group, if one is set.

A request makes at most 6 provider attempts in total, across models and fallback groups, and stops when the request timeout runs out.

Retryable failures are HTTP 408, 409 and 429, any 5xx except 501, timeouts before the provider responds, and connection failures. Retries back off exponentially with jitter, starting at 250 ms and capped at 8 s. A provider Retry-After header is honoured; when it asks for longer than the time left, the model is benched until then instead of waiting.

Requests the provider rejects on their merits (HTTP 400, 413 and 422) are not retried, not sent to another model, and do not count towards benching, because every model would reject them the same way. The provider's error is returned to the caller. The prompt-too-long and content-refused fallbacks below still apply.

What the timeout covers. The request timeout bounds the attempts up to the provider's response headers, including a call that connects and then goes quiet. It does not limit how long a stream may run once it has started.

Streaming. A streamed request can be retried or failed over until the first byte reaches the client. After that it is committed to the model that is streaming.

When everything fails, the caller receives the last provider's error with its status code, or 502 upstream_unreachable or 504 upstream_timeout when no provider answered. The trace records the refusal as all_deployments_failed.

Every response that reached a provider carries an x-ants-attempts header listing each attempt: the deployment, its outcome (ok, retryable, failed, timeout or benched), its duration, and the provider's status and message where there was one. The same list is on the trace.

Benching and cooling-off

When a model reaches Failures before benching within the Failure window, it is benched for the Cooling-off period. Every gateway replica skips it within about a second.

• Only real request failures count. Caller errors (400, 413, 422) and client disconnects do not.
• If every model in a group is benched, the one closest to the end of its cooling-off is tried anyway, so a cooling-off period never becomes an outage on its own.
• Benched models are shown on the Overview Model health card.

Fallback groups

In the Routing panel, a group can name up to three fallback groups in the same project:

Prompt-too-long and content-refused are recognised only from provider responses with status 400, 403, 413 or 422 whose error mentions a context-length or content-filter reason. A 429 or 503 is always treated as an ordinary retryable failure.

A group cannot fall back to itself, and a fallback that would form a loop is refused when you save it. A request passes through at most three fallback hops. When a fallback group serves the request, the x-ants-model-group response header names that group, and the trace records gateway.fallback_kind as general, context or content.

Requests that name no model

A key granted exactly one model group can leave model out of the request body, or send it empty. The request goes to that group, so an agent configured with only a base URL and a key still routes. A key granted two or more groups must name one; otherwise the request is refused with 403 model_not_allowed and a message listing the groups it may call. This applies to chat completions and embeddings.

Embeddings

POST /v1/embeddings addresses the same model groups and follows the same retry, failover, benching and fallback rules. Models whose provider family has no embeddings API (Azure OpenAI, Anthropic, AWS Bedrock and Google Vertex AI in the current release) are skipped, so a mixed group still serves embeddings from its OpenAI-compatible models. A group with no model that offers embeddings is refused with 400 embeddings_not_supported.

Response cache

Each group can answer identical chat requests from a cache. In the Routing panel, under Response cache, turn on Answer identical requests from cache and set Cache lifetime (seconds) (default 300, up to 86400).

• A request is identical when its messages, tools and generation parameters (such as temperature, max_tokens, response_format and seed) match. The cache is shared by every key that can call the group.
• A hit is served only after the key, model access, rate limits and budgets have been checked. It makes no provider call and costs nothing.
• Hits carry the header x-ants-cache: hit and x-ants-cost: 0, and the trace carries gateway.cache = hit. A streaming caller receives the cached answer as a stream.
• Errors, refusals and streams that did not finish are never cached.
• Keys in usage-only recording mode can read from the cache but never write to it.
• The cache applies to chat completions only, not embeddings.