Getting started with OpenTelemetry

This guide covers how to enable OpenTelemetry in Posit Connect, common configuration patterns, and how to use OpenTelemetry alongside Prometheus.

Minimal setup

Enable OpenTelemetry in your rstudio-connect.gcfg:

[OpenTelemetry]
Enabled = true

With this minimal configuration, Connect collects signals and persists them to disk. For details about its impact and retention, see Disk persistence.

Export to an observability backend

To send signals to your observability platform’s locally running agent, add an [OTLPEndpoint] section:

[OpenTelemetry]
Enabled = true

[OTLPEndpoint "mybackend"]
Endpoint = http://<host>:4318

The Endpoint setting is the base URL of an OpenTelemetry Protocol (OTLP) HTTP receiver. Port 4318 is the default OTLP HTTP port used by most collectors. Connect appends the signal-specific paths (/v1/traces, /v1/logs, /v1/metrics) automatically so do not include them in the URL.

By default, Connect sends all three signal types (logs, traces, and metrics) to each endpoint. You can configure each endpoint to receive only specific signal types:

[OTLPEndpoint "mybackend"]
Endpoint = http://<host>:4318
Logs = true
Traces = false
Metrics = false

You can configure multiple named endpoints to fan out signals to different observability platforms.

Most APM platforms require a locally running agent or collector to receive OTLP data. For example, Datadog requires the Datadog Agent running on the same host, which acts as the local collector and handles authentication to the Datadog backend. Ensure your platform’s collector agent is installed and running before configuring the endpoint.

Multiple instances on a shared cluster

When you run several Connect instances on the same Kubernetes cluster—for example, in separate namespaces—their signals arrive at your observability backend together. You cannot tell instances apart by service.name, because it identifies the process type rather than the instance: the server reports posit-connect, content jobs report posit-connect-content-<guid>, and package builders report posit-connect-python or posit-connect-nodejs. Every instance emits the same set of values. To group and distinguish them, Connect stamps two resource attributes that you control on its signals: service.namespace for the deployment or environment, and connect.node.name for the individual replica.

Group signals by deployment with a service namespace

Set OpenTelemetry.ServiceNamespace to a value that identifies the deployment or environment. Connect stamps it as the service.namespace resource attribute on every signal it emits—server, content jobs, and package builders—so all signals from one deployment share a single, stable value that you can group and filter by:

[OpenTelemetry]
Enabled = true
ServiceNamespace = data-science

Set the value per deployment or environment. When you set nothing, service.namespace defaults to posit-connect-ns.

Note

service.namespace and k8s.namespace.name are different concepts, even when they happen to hold the same value. service.namespace is an application-level, logical grouping from the OpenTelemetry semantic conventions: it groups related software components—for example, separating a billing application from an inventory application—regardless of where they run. k8s.namespace.name is an infrastructure-location attribute that records which Kubernetes namespace partition runs the pod. Set service.namespace to whatever logical grouping suits your deployment; it does not have to match your Kubernetes namespace.

Most APM backends promote service.namespace to a queryable label on their own, so this requires no extra collector configuration.

Label individual replicas with a node name

Within a deployment, connect.node.name distinguishes individual replicas. Connect derives it from Server.NodeName, stamping it on the server’s signals and propagating it to content job signals. In a Kubernetes deployment, NodeName defaults to the pod name (for example, rstudio-connect-7d9f8b6c5-x2k4p)—unique per replica, but ephemeral, because it changes on every rollout, restart, or rescheduling. That is useful for drilling into a specific replica, but prefer service.namespace for stable deployment-level grouping. Outside Kubernetes, set a distinct Server.NodeName per node:

[Server]
NodeName = connect-team-a

Filter by Kubernetes metadata

Connect does not detect Kubernetes metadata on its own, but its server honors the standard OTEL_RESOURCE_ATTRIBUTES environment variable: any resource attributes you set there are stamped onto the server’s signals. Set it on the Connect pod using the Kubernetes Downward API to tag them with the namespace, pod, and node:

env:
  - name: POD_NAMESPACE
    valueFrom:
      fieldRef:
        fieldPath: metadata.namespace
  - name: POD_NAME
    valueFrom:
      fieldRef:
        fieldPath: metadata.name
  - name: NODE_NAME
    valueFrom:
      fieldRef:
        fieldPath: spec.nodeName
  - name: OTEL_RESOURCE_ATTRIBUTES
    value: k8s.namespace.name=$(POD_NAMESPACE),k8s.pod.name=$(POD_NAME),k8s.node.name=$(NODE_NAME)

Most APM backends map these standard attributes to their own conventions. Datadog’s OTLP pipeline, for example, maps them as follows:

OpenTelemetry attribute	Datadog tag
`k8s.namespace.name`	`kube_namespace`
`k8s.pod.name`	`pod_name`
`k8s.node.name`	`kube_node`

You can then distinguish instances in different namespaces by kube_namespace, independent of service.name.

These attributes describe the Connect server’s own pod. Under off-host execution, package builds run in their own pods; their signals carry k8s.namespace.name (the build namespace) and service.namespace, but not k8s.pod.name or k8s.node.name, which Kubernetes assigns at schedule time.

Best practices

Service namespace: Set OpenTelemetry.ServiceNamespace per deployment, or let the Helm chart populate it from the Kubernetes namespace. This is the most reliable way to group and separate deployments, because Connect stamps it on server, content, and builder signals alike, independent of your collector’s Kubernetes enrichment.
Node name: Use connect.node.name (from Server.NodeName) to drill into individual replicas within a deployment. In Kubernetes it defaults to the ephemeral pod name; set it explicitly outside Kubernetes.
Collector config directory: Each replica needs its own private copy of CollectorConfigDir, where the embedded collector writes its runtime config, log file, and an auto-generated Transport Layer Security (TLS) certificate (default /etc/rstudio-connect/otelcol). Connect binds this certificate to the replica’s own network address and regenerates it on each start, so replicas must never share this directory. In Kubernetes, the default path is the pod’s own ephemeral (container-local) filesystem, which is already private to each replica—leave it there. Do not move it onto the shared volume that backs Server.DataDir, and do not back it with a hostPath volume, because replicas scheduled onto the same node would then share it.
Metric cardinality: Metric names are identical across instances—Connect does not prefix them per instance—so only resource attributes such as service.namespace and connect.node.name distinguish instances. Running many instances, or adding high-cardinality labels on top, multiplies the number of time series your backend stores. Prefer a small set of stable labels such as service.namespace over per-pod identifiers when you build dashboards and alerts.

Disk persistence

By default, Connect persists signals to disk as JSONL files. Diagnostic bundles include these files, which are useful for debugging. See the configuration reference for the full list of [OpenTelemetry] persistence settings: PersistenceEnabled, PersistenceDir, PersistenceMaxSizeMb, PersistenceNumBackups, and PersistenceMaxDays.

You can disable persistence, but we recommend keeping it on even if you export signals to an observability backend. This allows diagnostic bundles to include telemetry signals.

[OpenTelemetry]
Enabled = true
PersistenceEnabled = false

Disabling specific instrumentation

Connect enables both API and database instrumentation by default. You can selectively disable them:

[OpenTelemetry]
Enabled = true
APIInstrumentation = false       ; Disable API request tracing
DatabaseInstrumentation = false  ; Disable database metrics

Exporting logs via OpenTelemetry

You can export Connect’s server logs as structured OpenTelemetry logs. OpenTelemetry logs provide richer context through structured attributes (such as content GUIDs, user IDs, and request IDs) and automatic correlation with traces. Log scraping and OTel logs can also run side by side.

If your APM platform is already ingesting Connect’s structured logs from disk (for example, via a log-shipping agent), enabling OpenTelemetry log export sends the same data through a second path. You can disable disk-based log shipping in that case to avoid duplicate ingestion and extra costs.

To export logs via OpenTelemetry, configure an endpoint as follows. Logs defaults to true, but we set it in this example for clarity:

[OpenTelemetry]
Enabled = true

[OTLPEndpoint "mybackend"]
Endpoint = http://<host>:4318
Logs = true

Trace user enrichment

By default, trace spans carry content identifiers (such as content.guid) but not the identity of the user who owns that content. When TraceUserEnrichmentEnabled is set to true, Connect resolves user identity from span attributes before export, adding user.guid, user.username, and user.email to each span.

Connect resolves user identity using the following priority:

user.guid — If the span already carries a user.guid attribute, Connect looks up the corresponding username and email.
content.guid — If no user.guid is present but a content.guid attribute exists, Connect resolves the content owner and adds their user.guid, user.username, and user.email.

[OpenTelemetry]
Enabled = true
TraceUserEnrichmentEnabled = true

Note

Enrichment only applies to exported trace spans. Metrics and logs are not affected.

Prometheus and OpenTelemetry

Connect supports both Prometheus scraping and native OpenTelemetry export, and you can use both at the same time. We recommend native OpenTelemetry export for new setups because it provides richer context and automatic correlation across signals. If you already have a working Prometheus pipeline, Connect fully supports it.

Enabling OpenTelemetry with an existing Prometheus setup

If you already have Prometheus configured, you can enable OpenTelemetry without disrupting your existing Prometheus pipeline. When both are enabled and Metrics.PrometheusCollectorOTelEnabled = true, the Prometheus /metrics endpoint includes the new OpenTelemetry metrics along with the legacy Prometheus metrics.

[Metrics]
PrometheusListen = :3232
PrometheusCollectorOTelEnabled = true

[OpenTelemetry]
Enabled = true

Important

OpenTelemetry.Enabled must be true for OpenTelemetry metrics to appear on the Prometheus /metrics endpoint.

Migrating from legacy Prometheus HTTP metrics

When OpenTelemetry.Enabled = true and OpenTelemetry.APIInstrumentation = true, the legacy connect_http_* Prometheus HTTP metrics are automatically disabled and replaced by OpenTelemetry-standard http.server.* metrics. On the Prometheus /metrics endpoint, these appear with underscores and unit suffixes (for example, http.server.request.duration becomes http_server_request_duration_seconds).

The following table maps the legacy metrics to their OTel replacements as they appear on the Prometheus /metrics endpoint:

Legacy Prometheus metric	OTel replacement (Prometheus format)	Notes
`connect_http_request_inflight_gauge`	—	No OTel replacement. In-flight request tracking is not available with OTel HTTP instrumentation.
`connect_http_request_count`	—	Use `rate(http_server_request_duration_seconds_count[5m])` for request throughput.
`connect_http_request_duration_seconds`	`http_server_request_duration_seconds`	Label changes: `code` → `http_status_code`, `route` → `http_route`, `method` → `http_method`
`connect_http_request_size_bytes`	`http_server_request_body_size_bytes`	Label changes as above
`connect_http_response_size_bytes`	`http_server_response_body_size_bytes`	Label changes as above

Note

Non-HTTP legacy Prometheus metrics (queue, applications, build, content, and platform runtime metrics) remain unaffected by this change.