Docs/Llmops/Experiments

Experiments

An experiment runs something over every item in a dataset and records the result as a dataset run. Each item's result is a run item linked to a trace. Compare runs to see which prompt, model or version of your application does better on the same inputs.

There are three kinds:

Kind	What runs	Started from
Prompt experiment	A prompt version with a model from your LLM connections	The app, no code
Custom experiment	Your own application code	The Python SDK
Remote experiment	Your own service, called by a webhook	The app, after a one time setup. See Remote Experiments.

Select New experiment on a dataset, or Experiment on a prompt. The Run Experiment on Dataset dialog offers Prompt Experiment and Custom Experiment.

Prompt experiments

Prepare the prompt and dataset

The prompt needs at least one {{variable}} or message placeholder. Dataset item inputs must contain keys with the same names. For a prompt with {{question}}, an item input looks like {"question": "How do I reset my password?"}.

Fill in the form

In Prompt Experiment, select Create and fill in:

Experiment name (optional) and Description (optional)
Prompt: the prompt name and version
The model and its parameters, from your LLM connections
Dataset
Evaluators (optional): LLM-as-a-Judge evaluators that target experiment runs on this dataset. They score each result.

The form checks the configuration before you can run it. It reports an error if the prompt has no variables, if the dataset is empty or has only archived items, or if no item contains the prompt's variables.

Run

The experiment runs in the background over every active item. Each call uses your provider key and is billed by your provider.

Compare

Open the run from the dataset's runs table, or select several runs and choose Compare.

Custom experiments (SDK)

Run your own application over the items with the Python SDK. item.run(...) creates a run item for the item and links it to the current trace.

python

from ants_platform import get_client ants = get_client() dataset = ants.get_dataset("support/regression") for item in dataset.items: if item.status != "ACTIVE": continue with ants.start_as_current_span(name="regression-item", agent_name="support-agent"): with item.run(run_name="gpt-4o-mini-2026-10", run_description="Smaller model") as span: output = my_app(item.input) span.update(output=output) span.score_trace( name="exact_match", value=1.0 if output == item.expected_output else 0.0, ) ants.flush()

⚠

Open an observation with an agent_name around item.run(...), as shown. The Python SDK requires an agent name on the outermost observation of every trace, and item.run does not set one.

Use the same run_name for every item in one run. LLM-as-a-Judge evaluators that target experiment runs on this dataset also score run items created from the SDK.

You can run this script from CI. ANTS Platform records the scores; your CI job decides whether a result is good enough.

Remote experiments

If your experiment pipeline runs as a service, you can let your team start it from the dataset page. ANTS Platform sends a webhook with the dataset and a JSON config, and your service posts its results back as a dataset run. See Remote Experiments.

Comparing runs

The runs table on a dataset shows, per run, the number of run items, average latency, average total cost, aggregated run item scores and run-level scores.
Compare shows each item's output and scores for the selected runs side by side.
Every run item links to its trace for debugging.