Docs/Guides/Cost Optimization

Cost Optimization

Cost work follows a loop: record what every LLM call costs, find where the money goes, change something, and check that quality held. ANTS Platform covers the measuring and comparing; the changes themselves happen in your code or in Budget AI. Examples use the Python SDK (ants-platform). For other languages see the JS/TS SDK and Java SDK.

Record model and usage on every LLM call

Cost is calculated per generation from its model name and token usage, using the prices under Settings > Models. If you send a cost yourself, that cost is kept instead.

With the OpenAI drop-in client, model and usage are recorded for you:

python
from ants_platform import AntsPlatform from ants_platform.openai import OpenAI ants = AntsPlatform( public_key=os.environ["ANTS_PLATFORM_PUBLIC_KEY"], secret_key=os.environ["ANTS_PLATFORM_SECRET_KEY"], host=os.environ["ANTS_PLATFORM_HOST"], ) llm = OpenAI() def summarize(text: str, user_id: str) -> str: with ants.start_as_current_observation( name="summarizer", as_type="agent", agent_name="summarizer", input=text ) as agent: agent.update_trace(user_id=user_id, tags=["feature:summaries"]) completion = llm.chat.completions.create( name="summarize", model="gpt-4o-mini", max_tokens=200, messages=[{"role": "user", "content": f"Summarize in three sentences:\n{text}"}], ) summary = completion.choices[0].message.content agent.update(output=summary) return summary

For any other provider, record the generation yourself inside the agent observation and pass the usage the provider returns:

python
with ants.start_as_current_observation( name="summarize", as_type="generation", model="claude-3-5-haiku-20241022", input=prompt ) as generation: text, input_tokens, output_tokens = my_provider_call(prompt) generation.update( output=text, usage_details={"input": input_tokens, "output": output_tokens}, )

If you already know the price of a call, add cost_details={"input": 0.0012, "output": 0.0006} (USD) and it is used as reported.

Make sure every model has a price

Open Settings > Models and check that each model you use matches a definition. Managed definitions cover common OpenAI, Anthropic, Google and AWS Bedrock models. Add a custom definition for self-hosted, fine-tuned or new models, or clone a managed one to enter negotiated prices. The usage keys you send (input, output) must match the price keys. Price changes apply to generations ingested after the change. See Models & Pricing.

Attribute cost

Cost views can only split spend by what is on the trace. Set these on every trace:

FieldHow to set itWhere it splits cost
Trace nameThe root observation's nameTop 20 Use Cases (Trace) by Cost
Observation namename= on each generationTop 20 Use Cases (Observation) by Cost
Userupdate_trace(user_id=...)Top 20 Users by Cost; User consumption
Environmentenvironment= on the client or ANTS_PLATFORM_TRACING_ENVIRONMENTCost by Environment
Tags, release, version, metadataupdate_trace(tags=[...], version=..., metadata={...}); release= on the clientDashboard filters and custom dashboard breakdowns

For a different agent per trace, see Multi-Agent Systems.

Find where the money goes

QuestionWhere to look
Total spend, spend by model, user, feature or environmentGauge AI > Cost Dashboard. See Cost & Latency Dashboards.
How expensive does a single call or trace get?P 95 Cost per Trace, P 95 Input Cost per Observation and P 95 Output Cost per Observation on the same dashboard
Input versus output cost, tokens per modelGauge AI > Cost Performance Metrics, Model Usage tabs. See Cost & Performance Metrics.
Cost next to quality scoresCost Performance Metrics, Model costs and Scores
Trace and token volume over timeGauge AI > AI Usage. See AI Usage.
Cost per agentDiscover AI > AI Agent Center. See AI Agent Center.
One expensive requestDiscover AI > Tracing: open the trace and check each generation's input tokens, output tokens and cost

To keep a view of your own, clone the Cost Dashboard or build a custom dashboard with widgets over traces or observations.

Test a cheaper model with the AI Optimizer

Gauge AI > AI Optimizer replays a sample of an agent's real traces against up to three comparison models, scores the original and replayed outputs with the same LLM-as-a-Judge evaluator, and recommends a model only when quality holds. In the optimizer an agent is a trace name.

The optimizer reads the baseline model, cost (USD) and latency (milliseconds) from the model, cost and latency keys of the trace metadata. Add them on the root observation:

python
PRICES_PER_TOKEN = {"gpt-4o-mini": {"input": 0.15 / 1_000_000, "output": 0.60 / 1_000_000}} def summarize(text: str) -> str: with ants.start_as_current_observation( name="summarizer", as_type="agent", agent_name="summarizer", input=text ) as agent: started = time.monotonic() completion = llm.chat.completions.create( name="summarize", model="gpt-4o-mini", messages=[{"role": "user", "content": f"Summarize in three sentences:\n{text}"}], ) price = PRICES_PER_TOKEN["gpt-4o-mini"] cost = ( completion.usage.prompt_tokens * price["input"] + completion.usage.completion_tokens * price["output"] ) summary = completion.choices[0].message.content agent.update(output=summary) agent.update_trace( output=summary, metadata={ "model": "gpt-4o-mini", "cost": round(cost, 6), "latency": int((time.monotonic() - started) * 1000), }, ) return summary

Replace the prices with your own. Replay and judging run on your own provider keys from Settings > LLM Connections. See AI Optimizer and Recommendations.

Check quality after a change

Before and after switching a model or prompt, compare quality with LLM-as-a-Judge scores on live traces, or rerun a dataset as an experiment. Cost Performance Metrics shows the scores next to cost.

Reduce cost in your own code

These are changes you make in your application. Each one shows up in the cost views above once it ships.

  • Use smaller models for simple steps. Routing, classification and extraction often work on a small model. Keep the larger model for the final answer.
  • Trim prompts. Long system prompts, full conversation histories and too many retrieved documents drive input tokens. Check the input token count of your most expensive generations.
  • Cap output. Set max_tokens to what the step needs.
  • Cache repeated work. If the same question or document is processed often, store the result in your own cache and skip the LLM call.
  • Version prompts. Manage prompts in Prompts and link them to generations with prompt=, so you can compare cost and quality by prompt version.

Example of a simple in-process cache:

python
from functools import lru_cache @lru_cache(maxsize=1024) def classify(ticket_text: str) -> str: completion = llm.chat.completions.create( name="classify", model="gpt-4o-mini", max_tokens=5, messages=[{"role": "user", "content": f"Label as billing, bug or other:\n{ticket_text}"}], ) return completion.choices[0].message.content

Call it inside an agent observation. A cache hit makes no LLM call, so nothing is recorded for it.

Enforce budgets and limits with Budget AI

Observability shows spend after the fact; it has no budgets or cost alerts. To stop spend before it happens, send traffic through Budget AI, the AI gateway built into ANTS Platform. It requires the Budget AI entitlement (Enterprise).

NeedBudget AI feature
Monthly, weekly or daily spend capsBudgets in USD per key, tag, org unit or project. Stop requests refuses calls over the limit; Alert only records threshold crossings in the product. See Budgets & Limits.
Stop runaway loopsRequests per minute and tokens per minute per key and per project
Prefer the cheapest providerThe Lowest cost routing strategy picks the model in a group with the lowest estimated cost for the request. See Model Groups & Routing.
Keep working when a provider failsRetries and fallback groups
Avoid paying twice for the same requestThe response cache answers identical chat requests (same messages, tools and parameters) at no cost, for a lifetime you set (default 300 seconds)

Applications call Budget AI with any OpenAI client by changing the base URL and key:

python
from openai import OpenAI gateway = OpenAI( base_url=os.environ["ANTS_GATEWAY_BASE_URL"], api_key=os.environ["ANTS_GATEWAY_KEY"], ) response = gateway.chat.completions.create( model="prod-chat", messages=[{"role": "user", "content": "Hello"}], extra_headers={"langfuse-user-id": "user-1234"}, )

ANTS_GATEWAY_BASE_URL is your gateway address ending in /v1, shown under Project Settings > Budget AI. Every gateway request becomes a priced trace with the gateway tag. The Cost by Source and Cost by Gateway Key widgets on the Cost Dashboard split gateway spend from SDK traffic. See Getting Started and Observability & Cost.

ℹ

Use the plain openai client for gateway calls, as above. If you send gateway calls through the ants_platform.openai drop-in client, each call is recorded twice: once in your SDK trace and once as a gateway trace.

Common issues

SymptomFix
Generations have tokens but no costAdd or fix the model's price under Settings > Models; check that usage keys match the price keys.
Cost changed after a price update but old traces did notExpected. Prices apply to generations ingested after the change.
An agent is missing from the AI OptimizerIt needs at least 5 traces in the last 30 days and a model key in trace metadata.
AI Optimizer shows zero baseline costAdd cost and latency to the trace metadata.
Cost per user is emptySet user_id on the trace.

Next steps

© 2026 ANTS Platform, Inc.Docs v1.0 · Last updated June 2026