Evaluation & Credits
Every AI Optimizer run has an evaluation step. One LLM-as-a-Judge evaluator scores the baseline outputs (the answers already in your traces) and the outputs each comparison model produced during model replay. Because every model is judged by the same evaluator on the same snapshot of traces, the scores are directly comparable.
The judge runs on a model from your LLM connections, so it uses your own provider key and your provider bills you for it. On credit-based plans, the platform checks your organization's credit balance before the run starts and refuses the run if the balance cannot cover it.
Setting up the evaluator
After Save & Create Evaluator on the Select Models step, you configure the evaluator for the run:
- Evaluator template. Pick a managed template or one of your own. Managed templates use the project's default evaluation model; if none is set, you are asked to choose one first.
- Target filter. Prefilled with the agent (trace name) and the baseline model (
modelin trace metadata). You can add filters, but the agent and baseline filters stay. - Sampling. Sets the share of matching traces the run uses. The resulting trace count is shown next to the slider and is the
Nused for credits below. - Variable mapping. Maps the template variables, such as input and output, to trace fields. A preview shows the prompt the judge will receive.
The optimizer always evaluates existing traces, so the Evaluator runs on choice is hidden. Each judgement is stored as a numeric score on the trace, with the judge's reasoning as the score comment.
What gets scored
| Model | Traces scored | Score used for the recommendation |
|---|---|---|
| Baseline | The N snapshotted traces, as they are | Average score across the traces |
| Each comparison model | The replayed version of the same N traces | Average score across its replayed traces |
If Use batch inference model was on and there are at least 5 evaluations to run, the judge calls are sent through the judge provider's batch API (OpenAI, Azure OpenAI, Anthropic, Google AI Studio, Vertex AI or Bedrock). Otherwise they run in real time.
If every evaluation for a model fails, that model ends in an error state and gets no recommendation. When the project's ServiceNow integration has Enable Risk Records turned on, the platform also opens a high error rate risk record for that model.
Credits for an optimizer run
On credit-based plans, a run costs:
Nis the number of traces in the run (the count shown in the sampling section).Mis the number of comparison models (1 to 3).- Each baseline trace needs one judge score:
Ncredits. - Each replayed trace needs one generation and one judge score: 2 credits per trace per comparison model.
- Producing the recommendation itself is free.
For example, 100 traces against 3 comparison models need 100 + 600 = 700 credits.
The check happens when you submit the evaluator, before any model is called:
| Organization state | Result |
|---|---|
| Usage is blocked, or trial credits are exhausted | The run is refused. |
| Trial, or no payment method on file, and fewer credits than the run needs | The run is refused, and the message shows the credits needed and available. |
| Not on trial, with a payment method on file | The run starts. Credits beyond your balance are billed as overage. |
These credits are separate from what your LLM providers charge for the replay and judge calls. See Credits and Billing & Plans.
Changing a run
From the run's job page you can Edit an evaluator or Delete the whole job. Deleting cancels pending executions, removes the job's evaluator configurations and stops the run's remaining work.
Next steps
- Recommendations: how the scores turn into a recommendation.
- LLM-as-a-Judge: evaluator templates and judge models.
- Model Replay: how the comparison outputs are produced.