Skip to main content

AI Gateway

Enhanced Advanced Early Access

The AI Gateway is a part of Posit Connect that sends large language model (LLM) requests from your content to the provider. It supports these providers:

Requests to other LLM providers do not go through the AI Gateway, and Connect does not track or limit them.

Your administrator enables the AI Gateway. When it is enabled, Connect tracks AI spending for each user and each content item, and can limit that spending with budgets.

The AI Gateway handles the requests of some integrations. To learn which integrations use it, what changes for your code, and what Connect records about each request, see Integrations routed through the AI Gateway.

How Connect attributes requests

Connect attributes each request through the AI Gateway to a content item and to one user. Budgets and the AI Usage page use this attribution.

Requests from your content

A request from your content always counts against your content item. It also counts against one user:

Connect does not know the viewer for rendered or scheduled content. It also does not know the viewer if your content calls the provider with an HTTP client that does not send the viewer session token, unless the request uses an AWS integration with the Viewer authentication type.

If your content serves more than one viewer from the same process, your content must send each request with the session token of the viewer that the request is for. Connect cannot detect a token from the wrong viewer, so it attributes the request, and its budget usage, to that viewer.

If Connect does not know the viewer, all requests from your content count against the budget of the content owner. This includes requests that your content makes for other viewers. If your administrator refuses requests over a user budget, your content stops working for every viewer when the owner is over budget. To make each viewer use their own budget, use an AWS integration with the Viewer authentication type, or make sure that your content sends the viewer session token.

Requests from outside Connect

You can also use the AI Gateway from outside Connect. For example, Posit Assistant in a Posit Workbench session or a local coding tool can send requests through the AI Gateway after you sign in to Connect. To set up a local tool, see Use the AI Gateway from a local tool.

These requests count against your own user budget. They have no content item, so no content budget applies. On the AI Usage page, these requests appear when you show by User, and not when you show by Content.

Use the AI Gateway from a local tool

You can send requests from a coding tool on your computer, such as Claude Code, through the AI Gateway. Connect counts these requests against your user budget. See Requests from outside Connect.

A local tool can use an Anthropic integration or an AWS Viewer integration. Each kind of integration needs a different credential:

For both kinds, you need the AI Gateway URL of the integration. Your administrator gives you this URL. If the integration has an access control list (ACL), your administrator must also add you to it. If the integration is an AWS Viewer integration, they must also add you to the identity federation integration ACL.

Connect API keys for Anthropic integrations

The Anthropic route of the AI Gateway accepts only a short-lived Connect API key. It refuses the API keys that you create on your profile page.

A tool gets a short-lived key when you sign in through the Connect OAuth authorization server. The sign-in works like this:

  1. The tool registers with Connect at /oauth/v1/register.
  2. The tool starts a sign-in at /oauth/v1/device/authorize, and shows you a URL.
  3. You open the URL in a browser, sign in to Connect, and approve the request.
  4. The tool gets an API key and a refresh token from /oauth/v1/token. A refresh token lets the tool get a new key without a new sign-in.

These rules apply to the key and the refresh token:

  • The API key expires one hour after Connect creates it.
  • Before the key expires, the tool sends the refresh token to /oauth/v1/token. Connect returns a new API key and a new refresh token.
  • Each refresh token works one time, and expires after 30 days. If the refresh token expires, sign in again.

AWS credentials for AWS integrations

The Bedrock routes of the AI Gateway accept only temporary AWS credentials that Connect issued. They refuse Connect API keys. The tool signs each request with these credentials, in the same way that it signs a request to AWS. The AI Gateway checks the signature against the credentials that Connect issued.

Your administrator must set up these things first:

  • An AWS integration with the Viewer authentication type.
  • An integration with the OpenID Connect Federation template, which trusts your identity provider.
    • Your ID token must identify you with the same value as your Connect account, so you usually sign in to Connect through the same identity provider. See How Connect finds the user.
  • The Server.IdentityFederationExternalCredentialExchange setting must be enabled.

You must have these things:

  • A command that prints an ID token from your identity provider. The command depends on your identity provider. The ID token must have the audience that the federation integration requires. Ask your administrator which command to use.
  • A sign-in session for the AWS integration. To create it, open https://connect.example.com/__oauth__/integrations/<guid>/login in a browser, with the GUID of the AWS integration. Then sign in to the identity provider of the AWS integration.

To get credentials, the tool sends a POST request to /__api__/v1/oauth/integrations/credentials with these form fields:

Form fields for the AWS credential exchange.
Field Value
grant_type urn:ietf:params:oauth:grant-type:token-exchange
subject_token_type urn:ietf:params:oauth:token-type:id_token
subject_token The ID token from your identity provider
requested_token_type urn:ietf:params:aws:token-type:credentials
audience The GUID of the AWS integration

The access_token field of the response holds the credentials. It is Base64-encoded JSON with the fields accessKeyId, secretAccessKey, sessionToken, and expiration. The credentials expire at the time in expiration. Before they expire, the tool must get new credentials with a new ID token.

The tool must sign requests for the AWS region of the integration. If the region is different, the AI Gateway refuses the request.

AI Gateway URLs

Each integration has its own AI Gateway URL. The URL contains the route kind and the globally unique identifier (GUID) of the integration.

AI Gateway URLs.
Integration Route kind AI Gateway URL Credential
Anthropic anthropic https://connect.example.com/__gateway__/anthropic/<guid> Connect API key
AWS bedrock https://connect.example.com/__gateway__/bedrock/<guid> AWS credentials
AWS bedrock-mantle https://connect.example.com/__gateway__/bedrock-mantle/<guid> AWS credentials

Use the AI Gateway URL as the base URL of the provider API. The tool adds the API path to it. For example, Claude Code sends messages to https://connect.example.com/__gateway__/anthropic/<guid>/v1/messages.

Configure Claude Code

Claude Code can run a helper command to get a credential. In each example, replace https://connect.example.com with the URL of your Connect server, and <guid> with the GUID of the integration.

The Posit CLI gets and refreshes the Connect API key for you. To install it, see Installing the Posit CLI.

The posit connect token command needs rsconnect-python 1.33.0 or later. If you installed the Posit CLI before, upgrade it and its dependencies:

Terminal
uv tool upgrade posit-cli

If you installed the Posit CLI with pip, run pip install --upgrade posit-cli rsconnect-python in its virtual environment.

  1. Sign in to Connect with the Posit CLI:

    posit connect login https://connect.example.com

    If your browser is on a different computer, add --use-device-code. For more about the sign-in, see Using OAuth.

  2. Add these values to ~/.claude/settings.json:

    ~/.claude/settings.json
    {
  "apiKeyHelper": "posit connect token --refresh --server https://connect.example.com",
  "env": {
    "ANTHROPIC_BASE_URL": "https://connect.example.com/__gateway__/anthropic/<guid>",
    "CLAUDE_CODE_API_KEY_HELPER_TTL_MS": "3000000"
  }
}

  3. Start Claude Code.

Claude Code runs apiKeyHelper to get an API key. posit connect token --refresh uses the saved refresh token to get a new API key, and prints it. CLAUDE_CODE_API_KEY_HELPER_TTL_MS makes Claude Code run the command every 50 minutes, before the key expires after one hour.

If ANTHROPIC_API_KEY or ANTHROPIC_AUTH_TOKEN is set in your shell, Claude Code uses it in place of apiKeyHelper. Unset these variables before you start Claude Code.

If you do not use the Posit CLI for 30 days, the refresh token expires. Then posit connect token exits with status 4, and Claude Code cannot get a key. Run posit connect login again to fix this.

The script below gets AWS credentials for an AWS integration. It prints them in the format of the Claude Code awsCredentialExport setting. The script needs Python 3 on your computer. Before you start, get the items in AWS credentials for AWS integrations.

  1. Save this script as ~/bin/connect-gateway-aws:

    ~/bin/connect-gateway-aws
    #!/usr/bin/env python3
"""Print AWS credentials for the AI Gateway, in the awsCredentialExport format."""

import base64
import json
import os
import subprocess
import sys
import urllib.error
import urllib.parse
import urllib.request

SERVER = os.environ["CONNECT_SERVER"].rstrip("/")
INTEGRATION = os.environ["CONNECT_AWS_INTEGRATION"]
ID_TOKEN_COMMAND = os.environ["CONNECT_ID_TOKEN_COMMAND"]

# Get an ID token from your identity provider.
id_token = subprocess.run(
    ID_TOKEN_COMMAND, shell=True, check=True, stdout=subprocess.PIPE, text=True
).stdout.strip()

# Exchange the ID token for AWS credentials of the AWS integration.
body = urllib.parse.urlencode(
    {
        "grant_type": "urn:ietf:params:oauth:grant-type:token-exchange",
        "subject_token_type": "urn:ietf:params:oauth:token-type:id_token",
        "subject_token": id_token,
        "requested_token_type": "urn:ietf:params:aws:token-type:credentials",
        "audience": INTEGRATION,
    }
).encode()
req = urllib.request.Request(
    SERVER + "/__api__/v1/oauth/integrations/credentials",
    body,
    {"Content-Type": "application/x-www-form-urlencoded"},
)
try:
    with urllib.request.urlopen(req) as resp:
        token = json.load(resp)
except urllib.error.HTTPError as err:
    sys.exit(f"Credential exchange failed: {err.code} {err.read().decode()}")

creds = json.loads(base64.b64decode(token["access_token"]))
print(
    json.dumps(
        {
            "Credentials": {
                "AccessKeyId": creds["accessKeyId"],
                "SecretAccessKey": creds["secretAccessKey"],
                "SessionToken": creds["sessionToken"],
                "Expiration": creds["expiration"],
            }
        }
    )
)

  2. Make the script executable:

    chmod +x ~/bin/connect-gateway-aws

  3. Open https://connect.example.com/__oauth__/integrations/<guid>/login in a browser, and sign in.

  4. Add these values to ~/.claude/settings.json. Replace <id-token-command> with the command that prints an ID token, and <region> with the AWS region of the integration:

    ~/.claude/settings.json
    {
  "awsCredentialExport": "CONNECT_SERVER=https://connect.example.com CONNECT_AWS_INTEGRATION=<guid> CONNECT_ID_TOKEN_COMMAND='<id-token-command>' ~/bin/connect-gateway-aws",
  "env": {
    "CLAUDE_CODE_USE_BEDROCK": "1",
    "ANTHROPIC_BEDROCK_BASE_URL": "https://connect.example.com/__gateway__/bedrock/<guid>",
    "AWS_REGION": "<region>"
  }
}

  5. Start Claude Code.

Claude Code runs awsCredentialExport when it starts. It uses the credentials until five minutes before they expire, and then runs the command again.

Do not set CLAUDE_CODE_SKIP_BEDROCK_AUTH or AWS_BEARER_TOKEN_BEDROCK. The AI Gateway needs requests that are signed with the credentials from Connect.

If the exchange fails with an error about the sign-in session, open the integration login URL again and sign in.

Mantle is an Amazon Bedrock endpoint that serves Claude models with the Anthropic API. It uses the same AWS credentials as the Amazon Bedrock example. The IAM role of the AWS integration must allow bedrock-mantle:CreateInference.

  1. Do steps 1 to 3 of the Amazon Bedrock tab.

  2. Add these values to ~/.claude/settings.json. Replace <id-token-command>, <region>, and <model-id> with your values:

    ~/.claude/settings.json
    {
  "awsCredentialExport": "CONNECT_SERVER=https://connect.example.com CONNECT_AWS_INTEGRATION=<guid> CONNECT_ID_TOKEN_COMMAND='<id-token-command>' ~/bin/connect-gateway-aws",
  "env": {
    "CLAUDE_CODE_USE_MANTLE": "1",
    "ANTHROPIC_BEDROCK_MANTLE_BASE_URL": "https://connect.example.com/__gateway__/bedrock-mantle/<guid>/anthropic",
    "AWS_REGION": "<region>",
    "ANTHROPIC_MODEL": "<model-id>"
  }
}

  3. Start Claude Code.

Mantle model IDs start with anthropic. and have no version suffix, for example anthropic.claude-haiku-4-5. The models that you can use depend on your AWS account.

Do not set CLAUDE_CODE_SKIP_MANTLE_AUTH. The AI Gateway needs requests that are signed with the credentials from Connect.

Budgets

Your administrator can set AI spending budgets in US dollars for users and for content items. A budget applies to a period of one day, one week, or one month, and resets at 00:00 UTC.

Connect counts each request from your content against two budgets:

Default budgets and overrides

Your administrator sets a default budget for all users and a default budget for all content items. Your administrator can also set a different budget for one user or one content item. Only administrators can change budgets.

A budget is an amount in US dollars, or it is unlimited. If a budget is unlimited, Connect does not limit usage.

To see your budget and the budgets of your content, open the AI Usage page.

How Connect calculates usage

Connect calculates usage from the token counts of each request and the price of its model. Your administrator sets the prices of the models.

  • If a model has no price, its requests do not count against a budget. But if a budget is $0, every request is over the budget, including requests for a model with no price.
  • Connect calculates usage with the current prices. If your administrator changes a price, the usage of the current period changes too. This can put you or your content over a budget, or under it.

Connect checks the budgets before it sends a request to the provider, and counts the usage of the request after the response is complete. So the request that takes usage over a budget completes, and the budget applies to the requests after it.

When usage is over a budget

Your administrator decides what happens when usage is over a budget. Connect either allows the request, or refuses it with 429 Too Many Requests. The refusal message says when the budget resets. Make sure that your content handles this error.

Notifications

Connect sends in-app notifications when usage reaches part of a budget:

  • At 80% of the budget of a content item, Connect notifies the owner and collaborators of the content.
  • At 80% of the budget of a user, Connect notifies that user.
  • At 100% of either budget, Connect notifies the same people and the administrators.

Connect sends each notification one time in each budget period. Connect sends no notifications for a budget of $0.

When an administrator changes the budget of your user account, Connect notifies you. When an administrator changes the budget of a content item, Connect notifies its owner and collaborators.

View AI usage

The AI Usage page shows AI spending for the current budget period. To open it, go to the System page and select AI Usage. The page is available to publishers and administrators when the AI Gateway is enabled. Publishers see their own usage and the usage of content that they own or collaborate on. Administrators see all users and all content.

The top of the page shows three values:

  • Budget period: the length of the period and when it resets next.
  • Content over budget: what Connect does with a request from content that is over its budget. Warn means that Connect allows the request, and Block means that Connect refuses it.
  • User over budget: what Connect does with a request for a user who is over their budget.

Use Show by to switch between Content and User. To find an item, search by its name or its globally unique identifier (GUID). Select Refresh data to load the latest usage.

The table has these columns:

Columns on the AI Usage page.
Column Shows
Budget The budget in US dollars, or Unlimited if the budget has no limit.
Usage The usage in US dollars in the current period, and the percentage of the budget that it uses.
Cache hit The percentage of input tokens that the provider read from its prompt cache.
Tokens The total number of input and output tokens.
Open A link to the content item, which opens in a new tab, or to the profile of the user.