Docs/Batch Inference/Bedrock Setup # AWS Bedrock Batch Inference Setup
The [AI Optimizer](/docs/finops/ai-optimizer) can replay an agent's traces against a Bedrock comparison model through Amazon Bedrock batch inference instead of real-time calls. This page lists what to create in your AWS account and what to enter in the LLM connection.
Batch is turned on per optimizer run with **Use batch inference model** on the **Select Models** step. See [Model Replay](/docs/finops/model-replay) for how replay works for every provider.
## How it works
1. For each Bedrock comparison model, the platform collects the traces that have not already been replayed for that model.
2. If there are more than 100 of them, the platform creates a new S3 bucket in your account, in the connection's **AWS Region**, writes the requests to a JSONL file in it, and creates a Bedrock batch inference job that uses your service role to read the input and write the output. With 100 traces or fewer, that model is replayed in real time instead.
3. The platform checks the job status about once a minute. When the job is **Completed**, it reads the output from the bucket, stores the comparison results and runs the LLM-as-a-Judge scoring. If the job is **Failed**, **Stopped** or **Expired**, the model is marked as failed.
4. Jobs are checked for up to 2 days after they are submitted. Results of a job that finishes later are not collected.
Requests are sent in the Anthropic Claude Messages format, so use Anthropic Claude model IDs or inference profile IDs on Bedrock for batch. Each request uses `max_tokens` 1000 and temperature 0.7, and system messages are sent as user messages.
### What the platform creates in your account
<div className="dv2-table-scroll"><table><thead><tr><th>Resource</th><th>Name</th><th>Notes</th></tr></thead><tbody><tr><td>S3 bucket, one per batch job</td><td><code>finops-batch-<timestamp>-<project suffix>-<random></code></td><td>Created in the connection's region with a bucket policy that lets <code>bedrock.amazonaws.com</code> read and write it.</td></tr><tr><td>Input file</td><td><code>batch-input/<project id>/<job id>/<timestamp>-<random>.jsonl</code></td><td>Contains the trace inputs that are replayed.</td></tr><tr><td>Output folder</td><td><code>batch-output/<project id>/<job id>/<timestamp>-<random>/</code></td><td>Written by Bedrock. Contains the model outputs.</td></tr><tr><td>Bedrock batch job</td><td><code>finops-<project suffix>-<job suffix>-<timestamp></code></td><td>Visible in the Bedrock console under batch inference.</td></tr></tbody></table></div>
The platform does not delete these buckets or files. They hold your trace inputs and model outputs, so remove them according to your own retention policy, for example with an S3 lifecycle rule.
## Prerequisites
- An AWS account with Amazon Bedrock model access enabled for the Claude models you want to compare, in the region you will use.
- The models must support batch inference in that region. See [AWS: supported Regions and models for batch inference](https://docs.aws.amazon.com/bedrock/latest/userguide/batch-inference-supported.html).
- Permission to create an IAM role, an IAM user and an access key.
## Setup
<Steps>
### Step 1: Create the Bedrock service role
Bedrock assumes this role to read the input file and write the output. Create it with this trust policy (`trust-policy.json`):
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "bedrock.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
```
```bash
aws iam create-role \
--role-name bedrock-batch-inference-service-role \
--assume-role-policy-document file://trust-policy.json
```
When you save the connection, the platform checks that the trust policy allows `bedrock.amazonaws.com` to perform `sts:AssumeRole`.
### Step 2: Give the role access to the batch buckets
The buckets the platform creates start with `finops-batch-`. Attach an inline policy to the role (`role-s3-policy.json`):
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "S3Access",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::finops-batch-*",
"arn:aws:s3:::finops-batch-*/*"
]
}
]
}
```
```bash
aws iam put-role-policy \
--role-name bedrock-batch-inference-service-role \
--policy-name BedrockBatchS3Access \
--policy-document file://role-s3-policy.json
```
If you use a cross-Region inference profile, AWS may require more permissions on the role. See [AWS: create a service role for batch inference](https://docs.aws.amazon.com/bedrock/latest/userguide/batch-iam-sr.html).
### Step 3: Create the credentials the platform uses
The platform calls AWS with the access key you enter in the connection. Create an IAM user (or use an existing one) with this policy (`user-policy.json`). Replace `ACCOUNT_ID` with your AWS account ID.
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "BedrockInvokeAndBatch",
"Effect": "Allow",
"Action": [
"bedrock:InvokeModel",
"bedrock:ListFoundationModels",
"bedrock:ListInferenceProfiles",
"bedrock:CreateModelInvocationJob",
"bedrock:GetModelInvocationJob"
],
"Resource": "*"
},
{
"Sid": "BatchBuckets",
"Effect": "Allow",
"Action": [
"s3:CreateBucket",
"s3:PutBucketPolicy",
"s3:PutObject",
"s3:GetObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::finops-batch-*",
"arn:aws:s3:::finops-batch-*/*"
]
},
{
"Sid": "ValidationBucket",
"Effect": "Allow",
"Action": [
"s3:CreateBucket",
"s3:DeleteBucket"
],
"Resource": "arn:aws:s3:::ants-platform-bedrock-test-*"
},
{
"Sid": "ReadServiceRole",
"Effect": "Allow",
"Action": "iam:GetRole",
"Resource": "arn:aws:iam::ACCOUNT_ID:role/bedrock-batch-inference-service-role"
},
{
"Sid": "PassRoleForBedrock",
"Effect": "Allow",
"Action": "iam:PassRole",
"Resource": "arn:aws:iam::ACCOUNT_ID:role/bedrock-batch-inference-service-role",
"Condition": {
"StringEquals": {
"iam:PassedToService": "bedrock.amazonaws.com"
}
}
}
]
}
```
```bash
aws iam create-user --user-name ants-bedrock-batch
aws iam put-user-policy \
--user-name ants-bedrock-batch \
--policy-name AntsBedrockBatch \
--policy-document file://user-policy.json
aws iam create-access-key --user-name ants-bedrock-batch
```
Keep the access key ID and secret access key from the last command for Step 4.
### Step 4: Add the LLM connection
Go to **Project Settings > LLM Connections > Add LLM Connection** and fill in:
<div className="dv2-table-scroll"><table><thead><tr><th>Field</th><th>Value</th></tr></thead><tbody><tr><td><strong>LLM adapter</strong></td><td><code>bedrock</code></td></tr><tr><td><strong>Provider name</strong></td><td><code>bedrock</code></td></tr><tr><td><strong>AWS Region</strong></td><td>The region of your models, for example <code>us-east-1</code></td></tr><tr><td><strong>I intend to use Batch Inference for LLM optimizer</strong></td><td>Checked</td></tr><tr><td><strong>AWS Bedrock Service Role ARN</strong></td><td>The ARN of the role from Step 1, for example <code>arn:aws:iam::123456789012:role/bedrock-batch-inference-service-role</code>. This field appears when the checkbox is checked.</td></tr><tr><td><strong>AWS Access Key ID</strong></td><td>From Step 3</td></tr><tr><td><strong>AWS Secret Access Key</strong></td><td>From Step 3</td></tr><tr><td><strong>Custom models</strong></td><td>At least one Bedrock model ID or inference profile ID, for example <code>anthropic.claude-3-5-sonnet-20240620-v1:0</code></td></tr></tbody></table></div>
Click **Create connection**. See [LLM Connections](/docs/llmops/provider-keys) for the other fields.
The info icon next to **AWS Bedrock Service Role ARN** opens **Required IAM Role Permissions**, which shows the trust policy and the `iam:PassRole` statement from Steps 1 and 3. Scope the role's S3 permissions to the `finops-batch-*` buckets as shown in Step 2.
### Step 5: Turn on batch in the AI Optimizer
When you create an optimizer, check **Use batch inference model** on the **Select Models** step. See [AI Optimizer](/docs/finops/ai-optimizer).
</Steps>
<Callout type="info">
The platform looks up the Bedrock connection for batch jobs and their results by the provider name `bedrock`. Use exactly `bedrock` as the **Provider name** for the connection you use with batch inference.
</Callout>
## What is checked when you save
With **I intend to use Batch Inference for LLM optimizer** checked, **Create connection** runs these checks with your credentials before saving:
1. Every custom model ID exists in the region as a foundation model or an inference profile.
2. A temporary bucket named `ants-platform-bedrock-test-<timestamp>` can be created. It is deleted right away.
3. The role exists and its trust policy allows `bedrock.amazonaws.com` to assume it.
4. A short test message to the first custom model succeeds.
## Self-hosted deployments
On self-hosted ANTS Platform, **AWS Access Key ID** and **AWS Secret Access Key** are optional. When they are empty, real-time calls use the AWS SDK default credential provider chain (environment variables, the AWS credentials file, IAM roles for EC2 instances, IAM roles for ECS tasks).
Batch jobs do not use the default chain. To use Bedrock batch inference, enter an access key ID and secret access key in the connection.
## Troubleshooting
<div className="dv2-table-scroll"><table><thead><tr><th>Message or symptom</th><th>Cause</th></tr></thead><tbody><tr><td><code>AWS Role ARN is required when Batch Inference is enabled.</code></td><td>The checkbox is checked but <strong>AWS Bedrock Service Role ARN</strong> is empty.</td></tr><tr><td><code>Invalid AWS Role ARN format</code></td><td>The value does not look like <code>arn:aws:iam::ACCOUNT_ID:role/ROLE_NAME</code>.</td></tr><tr><td><code>Failed to validate IAM role: ... iam:GetRole permission</code></td><td>The role does not exist, or the access key cannot call <code>iam:GetRole</code> on it.</td></tr><tr><td><code>The IAM role ... does not allow Bedrock to assume it.</code></td><td>The trust policy is missing the statement from Step 1.</td></tr><tr><td><code>S3 bucket creation validation failed.</code></td><td>The access key cannot create or delete the <code>ants-platform-bedrock-test-*</code> bucket.</td></tr><tr><td><code>... do not exist in region ...</code></td><td>A custom model ID is wrong or not available in the selected region.</td></tr><tr><td>Bedrock model was replayed in real time</td><td>The run had 100 or fewer new traces for that model.</td></tr><tr><td>Batch job fails in the Bedrock console</td><td>Check that the role can read and write <code>finops-batch-*</code>, that the access key has <code>iam:PassRole</code> on the role, that the model supports batch in the region, and that the job is within your Bedrock Service Quotas for batch inference.</td></tr></tbody></table></div>
See [Batch Inference Limits](/docs/batch-inference/provider-limits) for the limits the platform applies.