OpenTelemetry signal reference

This reference catalogs all OpenTelemetry metrics and trace spans emitted by Posit Connect. Use this alongside the operational guide and health and performance guide to understand what signals are available for monitoring and troubleshooting.

Metrics

Application metrics

Metric Type Description Key Dimensions
application.count Gauge Running applications (deduplicated across workers) application.type
associations.count Gauge Content items associated with OAuth integrations integration.template, integration.auth_type
content.count Gauge Total number of content items content.type, runtime.language, runtime.version, access.type
integrations.count Gauge OAuth integrations integration.template, integration.auth_type
process.count UpDownCounter Running processes process.tag
schedule.count Gauge Schedules by status schedule.type, schedule.status (queued/running/success/failure)
session.ping.duration Histogram (ms) Session ping request duration —
users.active Gauge Distinct active users over time windows window (24h/7d/30d/1y), role (administrator/publisher/viewer)

Job and queue metrics

Metric Type Description Key Dimensions
job.cancelled Counter Jobs that were cancelled job.cancel.reason
job.completion Counter Completed jobs by status job.status (success/failure), job.exit_code
job.duration Histogram (s) Job execution duration. Buckets: [1, 5, 15, 30, 60, 300, 900, 1800, 3600, 10800, 21600, 43200, 86400] seconds job.status, job.exit_code
job.timeout Counter Jobs that timed out content.guid
queue.items.active_duration Gauge (s) Duration of oldest in-flight job queue.name, job.hostname
queue.items.age Gauge (s) Age of oldest item in queue queue.name
queue.items.completed Counter Queue items completed job.status, queue.name
queue.items.size Gauge Items in queue queue.name

Database metrics

Connection pool metrics

Metric Type Description
db.sql.connection.closed_max_idle Counter Connections closed due to SetMaxIdleConns
db.sql.connection.closed_max_idle_time Counter Connections closed due to SetConnMaxIdleTime
db.sql.connection.closed_max_lifetime Counter Connections closed due to SetConnMaxLifetime
db.sql.connection.max_open Gauge Maximum open connections to database
db.sql.connection.open Gauge Current open connections (idle and in-use)
db.sql.connection.wait Counter Total connections waited for
db.sql.connection.wait_duration Counter (ms) Total time blocked waiting for connections

Key dimensions: pool_name, db.system, status (idle/inuse for db.sql.connection.open)

Query latency metrics

Metric Type Description
db.sql.latency Histogram (ms) Database operation latency

Key dimensions: method, status

HTTP metrics

Metric Type Description Key Dimensions
http.server.request.body.size Histogram (bytes) HTTP request body size http.method, http.route, http.status_code
http.server.request.duration Histogram (s) HTTP request duration http.method, http.route, http.status_code
http.server.response.body.size Histogram (bytes) HTTP response body size http.method, http.route, http.status_code

Launcher metrics (off-host execution)

Metric Type Description Key Dimensions
launcher.client.request.duration Histogram (s) Launcher HTTP request duration including retries http.method, http.status_code
launcher.client.requests Counter HTTP requests to Launcher including retries http.method, http.status_code
launcher.client.retries Counter Retry attempts for Launcher requests http.method
launcher.client.retry.exhausted Counter Requests that exhausted all retries http.method

Worker pool metrics

Metric Type Description Key Dimensions
worker.pool.busy Gauge Workers currently processing jobs application.type
worker.pool.size Gauge Total workers available application.type
worker.pool.utilization Gauge Worker utilization ratio (0-1) application.type

License metrics

Metric Type Description Key Dimensions
license.expiration.days_remaining Gauge (days) Days until license expiration (negative when expired) —
license.shiny_users.current Gauge Concurrent Shiny users —
license.shiny_users.limit Gauge Licensed limit on concurrent Shiny users —
license.users.current Gauge Named users counting against license limit —
license.users.limit Gauge Licensed limit on named users —

Request rejection metrics

Metric Type Description Key Dimensions
requests.rejected Counter Requests rejected due to capacity or other limits rejection.reason (capacity/license/auth), rejection.type (shiny/api/content type)

Traces

Connect exports high-level lifecycle traces to your APM system, providing visibility into Connect operations. These traces show what Connect is doing without exposing internal implementation details.

See the operational guide for details on trace visibility.

HTTP and API spans

Span Name Description Key Attributes
api.service.request.handle Service request handling service.name
api.variant.render Variant render request content.guid, content.variant.key, execution.reason, execution.should_activate, execution.should_send_email
HTTP {method} {route} Automatic HTTP request tracing http.method, http.route, http.status_code, http.target

Schedule execution spans

Span Name Description Key Attributes
schedule.render Root span for schedule execution schedule.id, schedule.next_time, schedule.planned_run, schedule.delay_diff, schedule.is_stale, content.id, content.guid

Queue operations spans

Span Name Description Key Attributes
queue.item.enqueue Adding item to queue queue.name, queue.item.type
queue.item.process Queue consumer processing work item queue.item.id, queue.name, queue.item.type, queue.wait_time.ms, queue.processing_duration.ms, heartbeat.success_count, heartbeat.failure_count

Worker lifecycle spans

Span Name Description Key Attributes
worker.connection.await Waiting for worker to accept connections worker.id, worker.host, worker.port, worker.init_timeout.ms, worker.connection_duration.ms
worker.lifecycle Full worker lifecycle from creation to termination worker.id, content.guid
worker.process.startup Worker process startup runtime.type (python, shiny, plumber_api, tensorflow, container), worker.id, content.guid, job.key
worker.provision Worker provisioning and creation worker.id

Report execution spans

Span Name Description Key Attributes
report.execute Report execution report.type (quarto, jupyter, rmarkdown, rmarkdown_site), content.id, content.guid, content.bundle.id, execution.reason
report.process.start Report process startup (separates startup from execution) content.guid, job.key, runtime.type
report.setup Report setup and preparation report.type, content.id, content.guid

Launcher spans (off-host execution)

Span Name Description Key Attributes
launcher.job.start Launcher job setup and submission (Kubernetes) content.id, content.guid, content.variant.id, content.bundle.id, job.key, job.tag, job.cluster

Deployment spans

Span Name Description Key Attributes
deploy.content.launch Content deployment and launch content.id, content.guid, content.bundle.id

Email spans

Span Name Description Key Attributes
email.send Email send operation email.trigger, content.guid

Common span attributes

These attributes appear across multiple span types:

Attribute Description
content.bundle.id Bundle ID for content version
content.guid GUID of content item
content.owner.guid GUID of content owner
content.id Database ID of content item
content.name Short name of content
content.type / content.mode Content type (shiny, python_app, rmd, etc.)
content.variant.id Variant ID for content configuration
content.variant.key Human-readable variant key
email.trigger What triggered the email (e.g., report completion)
error.type Error type when operation fails
execution.reason Why content is executing (scheduled/manual/api)
heartbeat.failure_count Failed heartbeat count
heartbeat.success_count Successful heartbeat count
job.id Database ID of job record
job.key Unique job tracking key
job.tag Job type tag
launcher.job.exit_code Exit code when job completes
launcher.job.id Launcher’s job identifier
launcher.job.status Job status from launcher
process.pid Operating system process ID
queue.item.id Queue item database ID
queue.item.type Type of queued work
queue.name Queue identifier
queue.processing_duration.ms Time spent processing queue item
queue.wait_time.ms Time item waited in queue before processing
rebuild.required Whether package rebuild was needed
report.type Report format type: quarto, jupyter, rmarkdown, rmarkdown_site
runtime.type Runtime type: python, shiny, plumber_api, tensorflow, container
schedule.delay_diff Delay from planned time
schedule.id Schedule database ID
schedule.is_stale Whether schedule is stale
schedule.next_time Next scheduled run time
schedule.planned_run Originally planned run time
worker.connection_duration.ms Time taken to connect to worker
worker.host / worker.port Worker connection information
user.email User email address (added by trace user enrichment)
user.guid User GUID (added by trace user enrichment when resolved from content.guid)
user.username Username (added by trace user enrichment)
worker.id Unique worker identifier
worker.init_timeout.ms Configured worker initialization timeout

Span events

These events may appear within spans to mark significant moments:

Event Name Context
exception.appLocked Content is locked
exception.licenseInvalid License check failed
heartbeat.had_failures Heartbeat failures occurred during processing
noWorkFound No schedule ready to run
process started Worker process started successfully
rebuild.triggered Package rebuild was triggered
status.queued Item successfully queued
status.skipDuplicate Duplicate item not queued
workFound Schedule found and queued

Resource attributes

All signals include these resource attributes:

Attribute Description
connect.node.name Node name from configuration
host.name System hostname
service.name Always “posit-connect”

See also