LLM Gateway telemetry

Enhanced Advanced

When an integration carries the LLM Gateway trait, Posit Connect records telemetry about every request that passes through it. Connect derives everything from the request and response that already pass through the LLM Gateway, so publishers do not opt in and do not instrument anything.

Important

Read this page before routing content through the LLM Gateway. Capture is automatic, and by default it includes the full text of prompts and model responses. See Conversation content for what Connect records and Controlling capture for how to restrict it.

What is captured

Each request produces one span and contributes to the two conventional histograms, gen_ai.client.operation.duration and gen_ai.client.token.usage.

Connect follows the OpenTelemetry Generative AI semantic conventions. Refer to that specification for the meaning and value range of each gen_ai.* attribute. Every span carries the conventional attributes in three groups:

Request: model, max_tokens, temperature, top_p, and stop sequences.
Response: model, response ID, finish reasons, and input and output token counts.
Call: operation name, provider name, upstream address and port, and error.type when the provider returned a 4xx or 5xx.

Connect adds four attributes beyond the conventions:

Attributes Connect adds beyond the Gen AI semantic conventions.
Attribute	Description
`content.guid`	The globally unique identifier (GUID) of the content item that made the request. Span attribute only.
`gen_gateway.account`	Which account configured in the LLM Gateway served the request, so you can split usage across accounts.
`gen_ai.usage.cache_creation.input_tokens`	Tokens written to the prompt cache. An Anthropic extension to the conventions. Span attribute only.
`gen_ai.usage.cache_read.input_tokens`	Tokens read from the prompt cache. An Anthropic extension to the conventions. Span attribute only.

content.guid is the attribution key. Connect resolves it from the authenticated caller rather than from anything the request supplies, so content cannot spoof it. Connect excludes it from metrics to keep metric cardinality bounded.

Conversation content

By default, the LLM Gateway records the content of every conversation it proxies: the prompts your content sends, the responses the model returns, and the system instructions in between. This includes tool call arguments and tool results, and any data, identifier, or secret that appears inside message content.

Conversation attributes captured on each span.
Attribute	Contains
`gen_ai.system_instructions`	The system prompt sent with the request.
`gen_ai.input.messages`	The full input conversation, oldest message first.
`gen_ai.output.messages`	The messages the model returned, each with its finish reason.

Warning

Treat captured conversations as sensitive data. If your content puts customer records, credentials, or personal information into a prompt, Connect records that content on the span. The controls below decide where it can travel, not whether Connect records it.

Controlling capture

Two server-level settings govern this. Both are administrator-controlled, and publishers cannot opt individual content items in or out.

Settings that control conversation capture.
Setting	Default	Effect
`OpenTelemetry.ConversationCapture`	`full`	`full` records conversation content and exports it unredacted to configured external OpenTelemetry Protocol (OTLP) endpoints. `internal` records it and keeps it visible inside Connect to the content owner and collaborators, but the embedded collector masks every content attribute with `__REDACTED__` before export. `off` never records conversation content anywhere.
`OpenTelemetry.ConversationMaxBytes`	`16384`	Maximum serialized size of each conversation attribute, applied independently per attribute per span. A value of `0` or less means uncapped. Connect rejects a positive value below `256` at startup.

/etc/rstudio-connect/rstudio-connect.gcfg

[OpenTelemetry]
Enabled = true
ConversationCapture = internal

Choose internal when your external endpoints are not approved to receive prompt and response content. Choose off where conversation history must not be retained at all.

Important

ConversationCapture interacts with OpenTelemetry.PublicTracesOnly, which is enabled by default. Under that default, LLM Gateway Gen AI spans are not public and never leave the host. The difference between full and internal therefore applies only to gen_ai.* content on exported spans. Setting internal does not reduce what Connect stores locally. Only off does that.

When a conversation attribute exceeds ConversationMaxBytes, Connect shrinks it rather than dropping the span. For the message lists, it first drops whole messages, oldest first; only if that is not enough does it truncate the remaining fields and mark the result with a …[truncated] suffix. A span can therefore be missing early messages without carrying that marker. Truncation can also turn a JSON object into a bare JSON string, because cutting bytes out of a JSON object produces invalid JSON. Consumers that parse these attributes must tolerate both.

What is not captured

Viewer identity: for a Viewer AWS integration, Connect knows which viewer the credentials were issued for, but does not place that identity on the span. Attribution is at the content level, through content.guid.
Credentials: neither the real provider credential the LLM Gateway substitutes on the outbound leg, nor the Connect credential or AWS signature of the caller, appears in spans, metrics, or request logs.
Plain logs: Connect never writes conversation content to log files. It exists only as span attributes.

Note

“Not captured” means the LLM Gateway does not record it. It says nothing about what your provider records, or about telemetry your own content emits through content observability, which is a separate, publisher-opt-in feature.

Where the data goes

LLM Gateway telemetry follows the standard OpenTelemetry path in Connect, with no separate export mechanism and no separate storage. Connect exports nothing unless OpenTelemetry.Enabled is true.

Where Connect sends LLM Gateway telemetry, and how it treats conversation content on each path.
Destination	Configured by	Conversation content
On-disk signal files	`OpenTelemetry.PersistenceEnabled`, which is on by default	Unredacted. Connect never masks the local file copy.
Your Application Performance Monitoring (APM) platform	One or more `[OTLPEndpoint]` sections	Unredacted under `ConversationCapture = full`, masked as `__REDACTED__` under `internal`, and absent under `off`.

To send LLM Gateway telemetry to your existing observability platform, point an OTLP endpoint at it:

/etc/rstudio-connect/rstudio-connect.gcfg

[OpenTelemetry]
Enabled = true

[OTLPEndpoint "mybackend"]
Endpoint = http://collector:4318
Traces = true
Metrics = true

LLM Gateway signals are not separately routable. Any endpoint configured for traces receives LLM Gateway spans along with the other Connect spans. Filter on gen_ai.provider.name or the connect/gateway instrumentation scope in your backend to isolate them.

Warning

Under the default ConversationCapture = full, anyone with access to your configured observability backend can read the full text of prompts and model responses, not just usage counts. A Connect diagnostic bundle contains the unredacted local copy regardless of the redaction setting. Treat both as systems that hold conversation content, and restrict access accordingly.

Who can see what

Visibility of LLM Gateway telemetry.
Audience	Sees
Connect administrators	Everything, including unredacted conversation content, through the diagnostic bundle and the on-disk signal files.
Content owners and collaborators	Conversation content for their own content, when `ConversationCapture` is `full` or `internal`.
Observability platform users	Whatever Connect exports to that platform, subject to its own access controls. Exports include conversation content under `full` and mask it under `internal`.
Content viewers	Nothing.

Failure handling

Telemetry extraction never breaks a proxied request. Parsing is best-effort: attributes the parser cannot determine are absent from the span rather than recorded as zero, and Connect forwards the response bytes to your content untouched. Connect still records failed requests. A 4xx or 5xx from the provider produces a span with error.type set and the span status marked as an error.