Docs/Llmops/Datasets

Datasets

A dataset is a named collection of test cases. Each item has an input, an optional expected output and optional metadata. You run experiments against a dataset to compare prompts, models or versions of your application on the same inputs.

ℹ

Datasets is not in the sidebar today. Open it at /project/<projectId>/datasets. Among the built-in roles, only Owner can create and edit datasets in the app today. See Members & Roles.

Create a dataset

Create it

On the Datasets page select New dataset. Enter a Name, an optional Description and optional Metadata (JSON). A / in the name groups datasets into folders, for example support/regression.

From code:

python

from ants_platform import get_client ants = get_client() ants.create_dataset( name="support/regression", description="Questions that broke the support agent", )

Add items

Add items in any of these ways:

By hand. On the dataset, select Create new dataset item and enter Input, Expected output and Metadata as JSON.
From CSV. Select Upload CSV, map each column to input, expected output, metadata or not mapped, check the preview and import. Files can be up to 10 MB.
From a trace. On a trace or observation, select Add to datasets. The input and output are copied into a new item, which keeps a link to its source trace.
From code or the API. Use the SDK or POST /api/public/dataset-items.

python

ants.create_dataset_item( dataset_name="support/regression", input={"question": "How do I reset my password?"}, expected_output={"answer": "Use Forgot password on the sign-in page."}, metadata={"category": "account"}, )

Run an experiment

Select New experiment on the dataset. See Experiments.

Item fields

Field	Required	Notes
`input`	Yes	Any JSON. For prompt experiments, its keys must match the prompt's variables.
`expected_output`	No	The ideal answer, used by evaluators and your own checks.
`metadata`	No	Any JSON, such as category or difficulty.
`source_trace_id`, `source_observation_id`	No	Set when the item was created from a trace.
`status`	No	`ACTIVE` or `ARCHIVED`. Prompt experiments skip archived items. In your own code, check `item.status`.

Pass id= to create_dataset_item to update an existing item instead of creating a new one.

Read a dataset in code

python

dataset = ants.get_dataset("support/regression") for item in dataset.items: print(item.id, item.input, item.expected_output)

To run your own application over the items and record the results as a run, see Custom experiments.

Runs and comparison

Each experiment creates a run on the dataset. The runs table shows, per run, the number of run items, average latency, average total cost, aggregated run item scores and run-level scores. Select runs and choose Compare to see outputs and scores for each item side by side. Each run item links to its trace.

Other actions

Duplicate dataset copies a dataset and its items.
Edit changes the name, description and metadata.
Delete a dataset, an item or a run. Deleting cannot be undone.
Datasets, items and runs are also available through the public API. See API Reference.

Next steps

Experiments

Run prompts or your own code against a dataset

LLM-as-a-Judge

Score experiment runs automatically

Human Annotation

Review traces before adding them to a dataset