LLM Gateway

Enhanced Advanced

The LLM Gateway is a trait that some integration definitions carry. For the combinations of template and authentication type it serves, Posit Connect routes the traffic of the integration through itself instead of letting content call the provider directly. The trait is not a separate feature to configure, and there is no such thing as an “LLM Gateway integration.”

Everything else about the integration is unchanged: the same provider registration, authentication types, access control lists, and credential delivery mechanism apply. The trait adds three things on top:

• Credential protection: for providers that front a static credential, the real provider credential never enters the content process.
• Attribution: Connect attributes every request to the content item that made it, without publishers instrumenting anything. See LLM Gateway telemetry.
• A single egress point: provider traffic leaves the network from Connect rather than from each content process.

Which integrations carry the trait

Integrations that carry the LLM Gateway trait.

Integration	Authentication type	LLM Gateway trait	Route kind
Anthropic	Environment variables	Yes	anthropic
AWS	Viewer	Yes	bedrock
AWS	Service Account	Yes	bedrock
AWS	Workload Identity	No	Not applicable
All others	Any	No	Not applicable

Content using an AWS Workload Identity integration holds credentials that the AWS SDK obtained on its own from a token file. Connect never sees those credentials and cannot verify them, so the LLM Gateway does not serve those integrations. They keep working exactly as they do without the LLM Gateway.

Note

The trait applies to the whole integration definition, not to individual content items. If the LLM Gateway serves an integration, Connect routes every content item associated with that integration through the LLM Gateway.

Enabling the LLM Gateway

The LLM Gateway is off by default. Enable it in the configuration file and restart Connect:

/etc/rstudio-connect/rstudio-connect.gcfg

[Gateway]
Enabled = true

The LLM Gateway also requires Applications.OAuthIntegrationsEnabled. Connect enables that setting by default. If you disable it, the LLM Gateway does not start, and Connect logs a warning at startup. Content still receives LLM Gateway routes in that state, so those requests fail. Either enable OAuth integrations or disable the LLM Gateway.

Connect builds the LLM Gateway URLs it injects into content from Server.EnvAddress, which defaults to Server.Address. If both are empty, Connect injects no LLM Gateway route. Content stays pointed at the provider directly with its real credential, exactly as when the LLM Gateway is disabled.

Content processes pick up LLM Gateway routing the next time they start. Attaching or detaching an integration served by the LLM Gateway restarts the processes of that content item, so the change takes effect without a redeploy. Creating, editing, and deleting integrations takes effect immediately, with no restart.

What Connect injects into content

When a publisher attaches an integration served by the LLM Gateway to their content, Connect sets these variables in the content process environment. They point the SDK of the provider at the LLM Gateway route, so the content requires no code changes.

Environment variables Connect injects for integrations served by the LLM Gateway.

Integration	Variable	Value
Anthropic	ANTHROPIC_BASE_URL	<server>/__gateway__/anthropic/<guid>
Anthropic	ANTHROPIC_API_KEY	An ephemeral Connect API key, in place of the real Anthropic key
AWS (Bedrock)	AWS_ENDPOINT_URL_BEDROCK_RUNTIME	<server>/__gateway__/bedrock/<guid>
AWS (Bedrock)	ANTHROPIC_BEDROCK_BASE_URL	<server>/__gateway__/bedrock/<guid>
AWS (Bedrock)	AWS_DEFAULT_REGION	The region configured on the integration

Each integration gets its own route, identified by the globally unique identifier (GUID) of the integration definition. Content-level environment variables take precedence over the injected values, so a publisher who sets ANTHROPIC_BASE_URL on their content overrides the LLM Gateway route for that content item.

Connect sets AWS_DEFAULT_REGION so that SDK clients construct without additional AWS configuration. The value is informational for the client: the LLM Gateway derives the outbound region from the integration definition, not from the request.

Caller authentication

The LLM Gateway authenticates every request itself, using one of two models depending on the provider. Neither requires anything of the publisher.

Anthropic, and any provider that fronts a static upstream credential, uses the ephemeral Connect API key that Connect injected at process start. The SDK sends that key in the header it normally uses for the provider key. The LLM Gateway validates it, then sets the real provider credential on the outbound leg and strips the Connect credential. The Connect key never reaches the provider, and the provider key never reaches the content. Only ephemeral keys can drive the LLM Gateway: user API keys and service tokens cannot, even valid administrator keys.

AWS Bedrock content receives no Connect credential. It fetches temporary AWS Security Token Service (STS) credentials from Connect and signs its requests with them, exactly as it does when calling AWS directly. The LLM Gateway verifies that signature against the credentials it issued, then re-signs the outbound request with those same credentials. Because the outbound leg carries the credentials of the caller, routing through the LLM Gateway does not change AWS CloudTrail attribution. AWS still attributes each call to the viewer or content item that the credentials represent. The LLM Gateway accepts only header-form signatures and rejects presigned, query-string authentication.

The LLM Gateway then checks that the integration is attached to at least one content item, and that the calling content item is among those associated with it. The LLM Gateway does not route an integration that exists but is attached to no content.

Provider-specific behavior

The LLM Gateway proxies the public Anthropic API, including both streaming (server-sent events) and non-streaming responses, and forwards response bytes to the content unchanged.

For AWS, the LLM Gateway proxies only Bedrock runtime operations for the region of the integration: model invocation and conversation calls, guardrail application, and asynchronous invocation. It rejects Bedrock control-plane calls, such as ListFoundationModels, with 404 Not Found. Those belong on the AWS bedrock service endpoint rather than bedrock-runtime. Connect overrides only the runtime endpoint, so an SDK client constructed for the control plane continues to reach AWS directly.

The LLM Gateway caps request bodies at 10 MiB. A request over the cap fails with 401 on a Bedrock route, because the LLM Gateway cannot verify a signature against a truncated body. On an Anthropic route, it fails with 502.

Denials and errors

LLM Gateway response codes.

Status	Meaning
401 Unauthorized	The credential of the caller is missing, invalid, expired, of the wrong type, or its signature did not verify.
403 Forbidden	The caller authenticated, but its content is not associated with this integration, or its credentials name a different integration.
404 Not Found	The provider recognized the route but not the request path, for example a Bedrock control-plane call.
421 Misdirected Request	No integration matches this route: the GUID does not exist, its template does not match the route kind, the LLM Gateway does not serve its authentication type, or it is attached to no content.
502 Bad Gateway	The provider failed to construct, or the upstream call failed.
503 Service Unavailable	Connect cannot read the integration from its database.

Authentication failures are deliberately indistinguishable from one another. An unknown key, an expired credential, and a bad signature all return the same generic error, and every “no matching integration” outcome returns 421. A caller therefore cannot probe for valid keys or integration GUIDs.

Every LLM Gateway request produces a log line with the remote address, method, path, matched provider, response status, and duration. Connect never logs provider credentials, caller credentials, or request bodies.

See also

• LLM Gateway telemetry: what the LLM Gateway captures about each request, and where it goes.
• Credential delivery: the AWS STS and ephemeral API key mechanisms that the two authentication models rest on.
• Integrations overview: traits in the context of the other integration component parts.