Performance Optimization
Optimizing an LLM application means finding the slow or expensive steps, changing the model, prompt or routing, and checking that quality holds. ANTS Platform measures each of these from your traces. The changes themselves happen in your code, your prompts or Budget AI.
There is no performance optimizer SDK, automatic batching, auto-scaling or load balancer for your application. ANTS Platform shows where time and money go and helps you test alternatives.
Find the problem
| Question | Where to look |
|---|---|
| Which traces, generations and spans are slowest? | Gauge AI > Latency & Analytics shows p50 to p99 latency per trace, generation and span name, and latency per model. See Cost & Performance Metrics. |
| Where does the time go inside one request? | Open the trace in Discover AI > Tracing and read the timeline of its observations. See Traces. |
| How is latency trending? | Gauge AI > Latency Dashboard. See Cost & Latency Dashboards. |
| Which models and agents cost the most? | Gauge AI > Cost Dashboard, Cost Performance Metrics and AI Usage. |
| Is cost driven by input or output tokens? | Cost Performance Metrics > Model Usage splits cost and units by usage type. |
| Did a prompt change make things slower or costlier? | The prompt's Metrics tab compares versions on median latency, tokens and cost. See Prompt Management. |
| Something specific to your team? | Build a custom dashboard, for example p95 latency by model or cost by tag. |
Accurate cost needs model prices. See Models & Pricing.
Fix it
- Try a cheaper or faster model. The AI Optimizer replays an agent's real traces against comparison models, scores the outputs with LLM-as-a-Judge, and recommends a switch only when quality holds.
- Shorten prompts. Trim instructions and context in a new prompt version, then compare versions in an experiment.
- Cache identical requests. In Budget AI, turn on the response cache for a model group to answer identical chat requests without a provider call. See Response cache.
- Route around slow or failing providers. Budget AI routing strategies include lowest latency, least busy and lowest cost, with retries, timeouts and fallback groups. See Model Groups & Routing.
- Cap spend. Budget AI budgets and rate limits stop runaway usage before the provider is called. See Budgets & Limits.
- Change your own code. Parallelize independent calls, stream responses, or cache results in your application. Traces show the effect.
Check that quality holds
Build a test set
Create a dataset from representative traces.
Compare
Run the current and the changed setup as experiments with LLM-as-a-Judge evaluators attached.
Watch production
Keep evaluators on live traces and watch scores next to cost and latency on Cost Performance Metrics and Latency & Analytics.
Next steps
Planned capabilities are listed on the roadmap.