Incident Response
When an AI agent slows down or starts failing, ANTS Platform helps you find the cause in the traces, get the right people involved, and keep requests flowing through another provider while you fix it.
ANTS Platform does not scale, restart or roll back your services. Remediation happens in your systems; the platform gives you the evidence and, for traffic through Budget AI, automatic failover between providers.
Triage
Confirm the scope
On Gauge AI → Latency Dashboard, check P 95 Latency by Model and P 95 Latency by Use Case for the time window. A jump on one model points to a provider; a jump on one trace name points to that agent.
Find failing requests
In Discover AI → Tracing, filter by Level ERROR, the affected trace name and the time range. The status message on the failing observation usually carries the provider error.
Inspect a trace
Open a failing or slow trace. The observation tree shows which step took the time or raised the error, with the model, input and output at that step. For Budget AI traffic, the Gateway panel shows the key, model group, deployment and every attempt.
Record the evidence
Copy the trace IDs into your incident ticket so others can open the same traces in the project.
Routing incidents
| Tool | What the platform sends | Docs |
|---|---|---|
| PagerDuty | Incidents from governance events (unapproved models, drift) and from prompt automations | PagerDuty |
| ServiceNow | Incidents from governance events, and risk records from the AI Optimizer, such as a high error rate when every evaluation for a model fails | ServiceNow |
| Slack and Jira | Governance notifications, and AI Optimizer recommendations you share | Slack, Jira |
Latency or error spikes in your traces do not open incidents on their own. To page on them, poll the Metrics API from your monitoring system. See Alerting & Notifications.
Keeping traffic flowing with Budget AI
If your applications call models through Budget AI, the gateway handles provider trouble at request time:
- Retries and timeouts for failed provider calls, set by the model group's routing policy.
- Cooling-off that takes a failing deployment out of rotation for a while.
- Fallback groups that send the request to a different model group when the first one fails.
- Rate limits and hard budgets that refuse runaway traffic before it reaches a provider.
Configure these under Budget AI → Models & Routing and Budget AI → Budgets & Limits. See Model Groups & Routing.
After the incident
- Compare latency and error views before and after the fix to confirm recovery.
- If a provider or model was the cause, run the AI Optimizer to test an alternative with the same quality.