Performance Optimization
Speeding up an AI agent means finding the step that takes the time and then changing it: a slower model, a long prompt, a slow tool or retrieval call, or a provider with high time to first token.
ℹ
The views below are under Gauge AI and need Gauge AI read access (Owner, Department Admin, Security & Compliance and Cost Analyst).
Find the bottleneck
| Question | Where to look |
|---|---|
| Which agents are slow? | Latency & Analytics → Trace latency percentiles (p50 to p99 per trace name). |
| Which step inside them is slow? | Generation latency percentiles and Span latency percentiles, then open a slow trace in Discover AI → Tracing to see the timeline. |
| Is the model the problem? | Latency & Analytics → Model latencies, and on the Latency Dashboard, P 95 Latency by Model and Avg Output Tokens Per Second by Model. |
| Do users wait long for the first token? | P 95 Time To First Token by Model and Avg Time To First Token by Prompt Name. |
| Are long outputs the cause? | Cost Performance Metrics → Model Usage → Units by type (output tokens). |
See Cost & Performance Metrics and Cost & Latency Dashboards for every view.
Fix it
- Try a faster model. Run the AI Optimizer without batch inference. It replays the agent's traces on up to three models and marks a model latency-optimized when its p95 latency is lower and its average quality score is at least the baseline's.
- Run steps in parallel. Spans that run one after another in the trace timeline are candidates for concurrency in your code.
- Shorten prompts and outputs. Fewer input tokens cut processing time, and fewer output tokens cut generation time.
- Stream responses. Streaming lowers the time users wait for the first token.
- Route and cache with Budget AI. For gateway traffic, Budget AI can route to faster deployments, fail over from a struggling provider and serve cached responses. See Model Groups & Routing.
Confirm the change
After a change, compare the same views for the same trace names before and after. For model changes, the AI Optimizer's recommendation details show per-trace latency for the original and the new model.