Self-Hosting & Configuration

On ANTS Cloud the gateway is operated for you. This page is for self-hosted deployments, where the gateway runs as its own service next to web and worker. It is a separate, internet-facing service and is off by default.

Requirements

• An Enterprise self-hosted license, which provides the llm-gateway entitlement.
• The gatewayAi feature flag enabled for the users who manage Budget AI, or LANGFUSE_ENABLE_EXPERIMENTAL_FEATURES=true. See What you need to turn it on.
• The same Postgres and Redis as the rest of the platform, the same event storage bucket that ingestion uses, and a running worker. The worker processes gateway traces and runs the budget and key maintenance jobs.
• The gateway must share SALT and ENCRYPTION_KEY with web. SALT hashes keys; changing it invalidates every issued key. ENCRYPTION_KEY (64 hex characters) opens provider secrets stored in the database.

Enable with Helm

The gateway ships as the gateway subchart of helm/ants-platform.

Helm values

The gateway image is built from gateway/Dockerfile. A deployment that leaves gateway.enabled: false does not need it.

Environment variables

The gateway validates its configuration at start-up and refuses to start when it is incomplete.

On web, set GATEWAY_PUBLIC_BASE_URL (through global.gateway.publicBaseUrl) and the same GATEWAY_SECRETS_* values.

Provider secret storage

By default, provider secrets are encrypted into the platform database. To store new credentials in AWS Secrets Manager instead:

Both web (which writes secrets) and the gateway (which reads them) must see the same store, region and prefix. Scope the IAM policy to the prefix and grant GetSecretValue, PutSecretValue, CreateSecret, DeleteSecret and RestoreSecret; without the last two, deleting a credential or moving it back out of the managed store leaves the secret behind. Existing database credentials are moved one at a time with Move in Models & Routing. See Where provider secrets are stored.

Client addresses behind proxies

Per-key allowed source networks are checked against the caller's address, which the gateway reads from X-Forwarded-For counting proxies from the right. Set trustedProxyHops to exactly the number of proxies in your path: for example 1 for a load balancer straight to the service, 2 for a load balancer then an ingress controller, 0 when connections reach the pod directly.

• Too high, and a caller can forge its own address and bypass network allow-lists.
• Too low, and the gateway sees a proxy's address, so every network-confined key is refused.

trustedProxyCidrs lists the proxies exactly instead and takes precedence when set.

Ports, health and metrics

The gateway listens twice:

/metrics reports per-project and per-key counts, so it is kept off the public port. It returns JSON with request counts by status, refusal and error counters by code, in-flight requests, gateway latency before the provider call (percentiles), attempt, retry, fallback and benching counts, budget exhaustion counts, health check results, telemetry recorded and failed counts, configuration statistics (projects, keys, last successful version read, pending projects) and process memory and event loop delay.

The chart's liveness probe uses /health and the readiness probe uses /ready, both on the admin port. /ready fails while the replica is draining or its configuration is stale, so the load balancer stops sending it traffic.

On SIGTERM the gateway marks itself not ready, lets in-flight requests and streams finish, and waits for pending trace recordings before exiting, within a shutdown deadline.

Configuration propagation and fail-closed behaviour

The request path never reads Postgres. Each replica holds an in-memory snapshot of every served project's keys, model groups, deployments, credentials and budgets.

• Every change in the product writes Postgres and its audit log entry in one transaction, then bumps the project's configuration version in Redis.
• Each replica polls the versions every second (GATEWAY_CONFIG_POLL_MS) and reloads changed projects. A blocked key, a removed grant or a disabled project stops working on every replica within about a second.
• Every 60 seconds (GATEWAY_CONFIG_FULL_RELOAD_MS) each replica reloads every enabled project. This picks up entitlement changes and any version bump that failed; the product says "Saved, but not yet in force" in that case.
• Fail closed: if no version read has succeeded within 5 seconds (GATEWAY_CONFIG_STALE_MS), or a project's pending change has not loaded within that time, requests for that project get 503 gateway_config_stale. When the version read itself is failing, every request is refused and /ready fails.

Dependencies

The gateway does not connect to ClickHouse.

The worker runs two Budget AI jobs: one every 10 seconds writes budget and org unit spend from Redis to Postgres, and one every minute blocks keys whose rotation grace window has ended.

Network policy

With gateway.networkPolicy.enabled: true, the gateway's egress is limited to DNS, the platform data namespace and the CIDRs in gateway.networkPolicy.allowedEgressCidrs on the ports in allowedEgressPorts (default 443). A compromised gateway holds provider secrets in memory, so this limits what it can reach. Before enabling it, list every provider endpoint, and when secrets are in AWS Secrets Manager the Secrets Manager and STS endpoints. A Postgres or Redis hosted outside the cluster must also be listed, with its port.

Scaling

Replicas are stateless; all shared state is in Redis. Scale with replicaCount or the autoscaler. Rate limits, budgets, reservations and benching are shared across replicas. The Lowest latency and Least busy strategies use measurements taken on each replica. Keep terminationGracePeriodSeconds long enough for your longest streamed completions.