Docs/Finops/Ai Optimizer

AI Optimizer

The AI Optimizer answers one FinOps question with data: can this agent move to a cheaper or faster model without hurting quality? It takes a sample of the agent's real traces, replays their inputs against up to three comparison models, scores the baseline outputs and the comparison outputs with the same LLM-as-a-Judge evaluator, and compares quality, cost and latency.

ℹ

Open it from Gauge AI → AI Optimizer. The page needs the finops-recommendations entitlement, which every current plan includes. Runs use credits on credit-based plans, and every model call (replay and judging) runs on your own LLM provider keys, so your provider bills you for those calls.

How it works

text

agent traces ─► snapshot of N traces ├─► baseline outputs ───────────────► LLM-as-a-Judge score └─► replay inputs on comparison models ─► LLM-as-a-Judge score │ quality + cost + p95 latency per model ◄┘ │ recommendation

Snapshot. When you start a run, the platform fixes the list of traces it will use, so new traffic during the run does not change the comparison. Traces that contain an observation with level ERROR are left out.
Model replay. The input of each trace is sent to every comparison model, in real time or through the provider's batch API. See Model Replay.
Evaluation. The evaluator you chose scores the original (baseline) outputs and every replayed output. See Evaluation & Credits.
Recommendation. Each comparison model is checked against the baseline. See Recommendations.

Before you start

An agent with traffic. In the AI Optimizer an agent is a trace name. The optimizer leaves its onboarding screen once at least one trace name has 5 or more traces in the last 30 days. Pick an agent with enough recent traces to give the comparison a meaningful sample.
Model, cost and latency on the trace. The optimizer reads the baseline model from the model key in trace metadata, and the baseline cost and latency from the cost (USD) and latency (milliseconds) keys in trace metadata. Traces without a model key do not appear as a baseline, and traces without cost or latency give the baseline a cost or latency of zero, which prevents a cost or latency recommendation.
An LLM connection for each comparison provider. Comparison models come from Settings → LLM Connections (see Provider API Keys). Supported providers are OpenAI, Azure OpenAI, Anthropic, Google AI Studio (Gemini), Google Vertex AI and AWS Bedrock.
A judge model. The evaluator runs on a model from your LLM connections. Managed evaluator templates use the project's default evaluation model.
Prices for every model. Replay cost is calculated from Models & Pricing. A comparison model without a price definition is costed at zero.

python

from ants_platform import get_client client = get_client() with client.start_as_current_span(name="support-request", agent_name="support-agent") as span: span.update_trace( name="support-agent", metadata={"model": "gpt-4o", "cost": 0.0042, "latency": 1830}, )

Access

Action	Who can do it
View the AI Optimizer, recommendations, analytics and history	Roles with Gauge AI read access, or the AI Optimizer (FinOps) sub-module at read: Owner, Department Admin, Security & Compliance, Cost Analyst
Create, edit and delete optimizer runs	Roles with Gauge AI write access, or the AI Optimizer (FinOps) sub-module at write: Owner, Department Admin, Cost Analyst

⚠

Creating an optimizer also lists the project's LLM connections and evaluator templates. Today only the Owner role can read those, so other roles with write access see no comparison models or templates. Ask an Owner to create the optimizer; anyone with read access can then follow its results. See Members & Roles.

Creating an optimizer

Start the wizard

On Gauge AI → AI Optimizer, click Create Optimizer. If no agent has enough traffic yet, the page shows an onboarding screen with the same button. Your plan can limit how many optimizers a project can have.

1. Select Agent

Pick the Agent Name (each option shows its trace count and average cost), give the run an Optimizer Name, and add an optional Description. Under Optimizer Type, Cost Optimizer and Latency Optimizer are always on. Click Next: Select Models.

2. Select Models

Baseline Model: the models found in the agent's trace metadata over the last 90 days, with trace count, average cost and average latency. The most used model is preselected.
Comparison Models: pick one to three models from your LLM connections, sorted from cheapest to most expensive by their configured price. Models that match the baseline are hidden. If you have no connection yet, Add LLM Connection takes you to Settings → LLM Connections.
Use batch inference model: send the replay through the provider's batch API. Batch can be up to 50% cheaper on the provider side, but results take longer and latency optimization is skipped for batch runs. See Model Replay.

Click Save & Create Evaluator.

3. Set up the evaluator

You land on evaluator setup with the agent and baseline model already applied as the Target filter. Choose an evaluator template, adjust the filter and Sampling to decide how many traces the run uses, and map the template variables. The number of matching traces is shown in the sampling section and drives the credit check. See Evaluation & Credits.

4. Follow the run

After you submit, the job page opens. A phase indicator moves through Pending, Trace Gen, Evaluation, Recommend and Complete, and each model shows its own progress. From the job page you can edit the evaluator or Delete the job, which cancels pending executions and removes its evaluator configurations.

Pages in the AI Optimizer

Tab or page	What it shows
Recommendations	Cost and latency recommendations per optimizer run, with projected savings and Jira, Slack and Linear actions. See Recommendations.
Analytics	Projected monthly and annual savings, a monthly savings trend, optimization status and per-agent analysis over the last 30 days.
Execution History	Past runs by agent and recent activity, with links to each job.
Job page	Configuration, evaluators and scores, per-model progress and status for one run.
Recommendation details	Per-trace cost, latency and score analysis for the original model and each comparison model.

Next steps

Model Replay: providers, batch inference and what is sent to your provider.
Evaluation & Credits: how quality is scored and how many credits a run uses.
Recommendations: how a model qualifies and how savings are projected.
Models & Pricing: keep candidate models priced so savings are accurate.