Docs/Llmops/Evaluations

LLM-as-a-Judge

LLM-as-a-Judge evaluators score your data using an LLM as the grader. An evaluator can run continuously on incoming traces, on traces you already have, or on experiment runs against a dataset. Each result is stored as a numeric score between 0 and 1, with the judge's one sentence reasoning saved as the score comment.

ℹ

Evaluators run on an LLM that you connect with your own provider API key. Calls to the judge model are billed by your provider. See Provider API Keys.

How it works

text

trace or experiment run item ──► evaluator (template + judge model) ──► score written back

An evaluator ties together a template, a judge model, a target, variable mappings, filters and a sampling rate. Matching items are queued, the judge model grades them, and the result is written as a score.

Key concepts

Concept	Meaning
Template	The judge prompt plus the output schema (a score and a one sentence reasoning). Use a managed template or create your own.
Judge model	The LLM that grades. Use the project's default evaluation model or pick a model for the template. Supported providers: OpenAI, Anthropic, Azure OpenAI, Amazon Bedrock, Google Vertex AI and Google AI Studio.
Target	Live tracing data (traces) or Experiment runs (dataset run items).
Variable mapping	Maps template variables such as `{{query}}` and `{{generation}}` to fields. For traces you can read the input, output or metadata of the trace or of a span, generation or event in it. For experiment runs you can also read the dataset item's input, expected output and metadata.
Filters	Limit which traces or run items are evaluated.
Sampling	The share of matching items to evaluate, to control cost and volume.
Delay	Seconds to wait before evaluating, so a trace can finish first.
Time scope	Evaluate new items as they arrive, existing items, or both.

Managed evaluators

ANTS Platform ships 19 managed LLM-as-a-Judge templates, listed below by their names in the app. Each returns a score between 0 and 1.

Group	Templates
Quality	Helpfulness, Relevance, Correctness, Conciseness
Safety	Hallucination, Toxicity
RAG and context	Contextrelevance, Contextcorrectness, Context Precision, Context Recall, Faithfulness
Answer checks	Answer Correctness, Answer Relevance, Answer Critic
Agents and tasks	Goal Accuracy, Topic Adherence Classification, Topic Adherence Refusal
Other	Simple Criteria, SQL Semantic Equivalence

Custom evaluators

You can create your own templates. Write the judge prompt with {{variable}} placeholders and describe what the score and reasoning should contain. Custom templates are shown as "User maintained" next to the managed ones.

You can also compute scores in your own code, for example rule based checks or your own models, and send them to ANTS Platform with the SDK or the POST /api/public/scores endpoint. See Scores.

Setting up an evaluator

Connect a judge model

Add an LLM connection with your provider key, then set the project's default evaluation model or choose a model on the template.

Choose a template

Pick a managed template or create a custom one.

Choose the target

Select Live tracing data to score traces, or Experiment runs to score dataset run items.

Map variables

Map each template variable to the field it should read, for example {{query}} to the trace input and {{generation}} to a generation's output.

Scope and sample

Add filters, set the sampling rate and an optional delay, and choose whether to evaluate new items, existing items, or both.

Save

Matching items are scored automatically. Results appear as scores on each trace or run item, and each evaluator has an execution log.

ℹ

Sampling keeps cost in check. Evaluating every trace with a strong judge model can be expensive. Start with a sample, confirm the scores are useful, then increase it.

Evaluating offline with datasets and experiments

Datasets hold test cases with an input, an optional expected output and metadata. You can create items in the UI, upload a CSV, or add items from existing traces.

You can run an experiment against a dataset in two ways:

Prompt experiment (UI). Run a prompt version and model configuration over every dataset item and select evaluators to score the results.
Custom experiment (SDK). Run your own application over the dataset items with the Python SDK. Each item run is linked to a trace and recorded as a dataset run item.

Evaluators that target Experiment runs score new run items automatically, including run items created from the SDK. You can compare runs side by side in the dataset view.

Because experiments run programmatically through the SDK, they can be run from CI via the SDK:

python

from ants_platform import AntsPlatform ants_platform = AntsPlatform() dataset = ants_platform.get_dataset("<dataset_name>") for item in dataset.items: with ants_platform.start_as_current_span(name="eval-item", agent_name="<agent_name>"): with item.run(run_name="<run_name>") as span: output = my_app(item.input) span.update_trace(input=item.input, output=output) span.score_trace( name="exact_match", value=1.0 if output == item.expected_output else 0.0, ) ants_platform.flush()

The Python SDK requires an agent_name on the outermost observation of every trace, so open a span with one around item.run(...).

⚠

ANTS Platform records evaluation results. It does not block deployments based on scores, and it does not include built-in red-teaming. If you want a release gate, read the scores in your own CI job and decide there.

Planned capabilities are listed on the roadmap.

Next steps

Scores: where evaluation results land.
Datasets: build test sets.
Experiments: offline evaluation against datasets.
Human Annotation: add human labels alongside judge scores.