Docs/Batch Inference/Provider Limits

Batch Inference Limits

Batch inference is used by the AI Optimizer. When you check Use batch inference model on the Select Models step, two parts of the run go through the providers' batch APIs instead of real-time calls:

Model replay: the traces are sent to each comparison model as one batch job per model. See Model Replay.
LLM-as-a-Judge scoring: the evaluations are sent to the judge model's provider as a batch when there are at least 5 of them. With fewer, or if the batch cannot be submitted, the evaluations run in real time.

Providers typically price batch requests lower than real-time requests, up to 50% less, in exchange for slower results.

Supported providers

LLM adapter	Batch API used	Setup needed	Applied by the platform
`openai`	OpenAI Batch API	API key	None
`azure`	Azure OpenAI batch	The model must be deployed with the Global Batch deployment type, otherwise the job fails	Deployment names are not checked for batch support
`anthropic`	Anthropic Message Batches API	API key	None
`google-ai-studio`	Gemini API batch mode	API key	None
`google-vertex-ai`	Vertex AI batch prediction	Vertex AI Setup	Google publisher (Gemini) models only
`bedrock`	Amazon Bedrock batch inference	Bedrock Setup	Batch is used only when more than 100 new traces are replayed for the model; Anthropic Claude request format

Limits applied by the platform

Limit	Value
Bedrock replay threshold	More than 100 traces that have not already been replayed for that model. With 100 or fewer, that model is replayed in real time.
LLM-as-a-Judge batch threshold	At least 5 evaluations. Below that, evaluations run in real time.
Jobs per comparison model	One batch job per model per run. All traces for the model go into that job; the platform does not split them.
Request settings	Each replay request uses `max_tokens` 1000 and temperature 0.7.
Status checks	Every 1 to 2 minutes, depending on the provider.
Result window	Jobs are checked for up to 2 days after submission. Results of a job that finishes later are not collected.
Latency recommendation	Not produced for batch runs. Only cost and quality are compared.

Traces that were already replayed for the same model in an earlier run are reused and are not sent again.

Model support check

When you check Use batch inference model and click Save & Create Evaluator, the platform checks the selected comparison models against a list of batch-capable models per adapter:

If a model is not on the list, you see Batch Inference Not Supported with the model names. Uncheck batch or pick other models.
If no list is configured for the deployment, you see Batch Inference Validation Skipped and the run continues without the check.
Adapters with no models on the list, such as azure, are not checked.

On self-hosted deployments the list comes from the BATCH_INFERENCE_MODELS_BY_ADAPTER environment variable of the web container: a JSON object that maps each adapter (for example bedrock, openai) to an array of model IDs. Bedrock IDs match with or without the us., eu. or apac. prefix.

Provider limits

Each provider sets its own limits for a batch job, such as the maximum number of requests, the maximum input file size, quotas per model and region, and for Amazon Bedrock a minimum and maximum number of records per job. Because the platform sends all of a model's traces in one job, a run with more traces than the provider allows fails for that model. Check the provider's current limits before large runs:

Choosing batch or real-time

	Batch	Real-time
Provider price	Usually lower	Standard
Speed	When the provider finishes the job, which can take hours	As each call returns
Latency recommendation	No	Yes
Minimum traces	Bedrock: more than 100. Other providers: none.	None
Best for	Large cost comparisons	Quick comparisons and latency optimization

⚠

If you need a latency recommendation, run the optimizer without batch inference.