LLMOps
LLMOps in ANTS Platform covers the work around the LLM calls your agents make: managing prompts, testing them, measuring quality and choosing models. Every feature on this page works on the traces your applications already send to ANTS Platform.
Features
| Feature | What it does |
|---|---|
| Prompt Management | Versioned text and chat prompts with labels such as production. Your app fetches them at runtime. |
| Prompts API & Caching | REST endpoints to fetch, list, create and relabel prompts, and how caching affects freshness. |
| Playground | Run prompts against models side by side and save the result as a prompt version. |
| LLM Connections | Your provider keys for OpenAI, Anthropic, Azure OpenAI, Amazon Bedrock, Google Vertex AI and Google AI Studio. Used by the Playground, experiments, evaluators and the AI Optimizer. |
| Schemas & Tools | Saved structured output schemas and tool definitions for the Playground. |
| Datasets | Test cases with an input, an optional expected output and metadata. |
| Experiments | Run a prompt, your own code, or a remote service over a dataset and compare runs. |
| Remote Experiments | Start your own experiment pipeline from the dataset page through a webhook. |
| LLM-as-a-Judge | Score traces and experiment runs with managed or custom judge templates. |
| Scores | Numeric, categorical and boolean quality scores from the SDK, the API, evaluators or people. |
| Human Annotation | Annotation queues for reviewing traces, observations and sessions. |
| Model Lifecycle | The AI Inventory catalog of models and frameworks, approvals, and detection of unapproved models. |
Related features outside this section:
- AI Optimizer replays an agent's traces against cheaper or faster models and recommends a switch when quality holds.
- Budget AI is the AI gateway: one OpenAI-compatible endpoint with keys, budgets, rate limits, routing and fallbacks.
- Models & Pricing defines how token usage is priced.
Where to find these features
Most LLMOps pages are not in the sidebar today. The pages still work, and you reach them like this:
| Feature | How to open it |
|---|---|
| Prompts | /project/<projectId>/prompts, or the prompt link on a generation in a trace |
| Playground | The Playground button on a prompt or on a generation in a trace, or /project/<projectId>/playground |
| Datasets | /project/<projectId>/datasets |
| LLM-as-a-Judge | /project/<projectId>/evals, or from the AI Optimizer flow |
| Scores | /project/<projectId>/scores |
| Human Annotation | /project/<projectId>/annotation-queues, or from the link shown after you add items to a queue |
| LLM Connections, Score Configs, Protected Prompt Labels, Models | Project Settings |
Replace <projectId> with the ID in your browser's address bar when you are inside a project.
Access to each feature depends on your role. See Members & Roles.
Getting data in
All of these features build on traces. Instrument your application with the Python SDK, the JS/TS SDK, a framework integration or OpenTelemetry. A minimal Python example with the OpenAI drop-in:
The client reads ANTS_PLATFORM_PUBLIC_KEY, ANTS_PLATFORM_SECRET_KEY and ANTS_PLATFORM_HOST from the environment. The Python SDK requires an agent_name on the outermost observation of each trace; nested observations inherit it. The OpenAI call is recorded as a generation with model, token usage, cost and latency.