Databricks connection patterns on Connect using workload identity

These access patterns enable you to connect to and read from Databricks databases or schemas in Posit Connect using Connect Workload Identity. They include both Python and R code examples for connecting to Databricks using Spark and SQL.

Choose a different pattern by referring to the table in Choose a connection for Databricks.

Not sure which pattern to use?

For an overview of credential types and scope in different environments, see Credential types and Managing data access from development to deployment.

About Connect workload identity

Administrator configuration required

Your Connect Administrator must configure a workload identity type Databricks integration in Connect. Your Databricks Workspace administrator must also configure a federation policy for Connect.

For detailed setup instructions, see the Databricks OAuth Integrations page in the Connect Admin Guide.

The Connect workload identity integration allows publishers to use the credentials of a service account (known as a service principal in Databricks) when accessing protected resources. The integration avoids the need to store long-lived credentials in Connect itself.

Every viewer will have the same experience when interacting with the content. Workload identity integrations support both interactive and rendered content types, along with access patterns using Anyone - no login required and Share links.

Required steps when publishing

You must enable the workload identity integration for your specified content using the Access tab in the Content Settings pane in Connect, or by selecting the integration upon deployment.

For more details, refer to the Adding Integrations section of the Connect User Guide.

Spark cluster connection

To authenticate with Databricks using workload identity, your code leverages the Databricks workload identity federation. The posit.connect.external.databricks module from the posit-sdk package handles authentication automatically using the workload identity configuration.

The code example below uses different authentication strategies depending on the environment:

DefaultCredentials performs Databricks workload identity federation when deployed to Connect: it reads the OIDC identity token that Connect provides and exchanges it for a Databricks access token belonging to the integration’s service principal
WorkbenchStrategy uses Workbench managed credentials in Workbench
databricks_cli provides a fallback for local development.

This approach allows the same code to run seamlessly across all three environments.

import os

import pandas as pd
from databricks.connect import DatabricksSession
from databricks.sdk.core import Config, databricks_cli
from databricks.sdk.credentials_provider import DefaultCredentials
from posit.connect.external.databricks import databricks_config
from posit.workbench.external.databricks import WorkbenchStrategy

# Environment variables
DATABRICKS_HOST_URL = f"https://{os.getenv('DATABRICKS_HOST')}"
CLUSTER_ID = os.getenv("DATABRICKS_CLUSTER_ID")

# Configure Databricks connection using environment variables.
# `DefaultCredentials()` is the Connect strategy for workload identity. On Connect,
# the Databricks Workload Identity integration provides an OIDC identity token, which
# the Databricks SDK federates into a Databricks access token for the integration's
# service principal—no stored secret or user-level token is required.
config = databricks_config(
    posit_default_strategy=databricks_cli,
    posit_workbench_strategy=WorkbenchStrategy(),
    posit_connect_strategy=DefaultCredentials(),
    host=DATABRICKS_HOST_URL,
    cluster_id=CLUSTER_ID,
)

# Create Databricks Spark session
spark = DatabricksSession.builder.sdkConfig(config).getOrCreate()

# Read a table located at "samples.nyctaxi.trips"
df = spark.read.table("samples.nyctaxi.trips")
df.show(5)

# Convert the Spark table to a Pandas DataFrame
pandas_df = df.limit(5).toPandas()

pandas_df

This connection method will be supported in a future release of sparklyr.

For more in-depth information and complete examples, see:

The Connect User Guide OAuth Integrations section for more detail on these concepts.
The Connect Cookbook for complete examples of using Databricks integrations in Connect.

SQL warehouse connection

The code example below uses different authentication strategies depending on the environment:

DefaultCredentials performs Databricks workload identity federation when deployed to Connect: it reads the OIDC identity token that Connect provides and exchanges it for a Databricks access token belonging to the integration’s service principal
WorkbenchStrategy uses Workbench managed credentials in Workbench
databricks_cli provides a fallback for local development.

This approach allows the same code to run seamlessly across all three environments.

import os

import dotenv
from dotenv import load_dotenv

load_dotenv()  # Load environment variables from .env file

import pandas as pd
from databricks.sdk.core import (
    databricks_cli,  # this can be removed if not using `posit_default_strategy`
)
from databricks.sdk.credentials_provider import DefaultCredentials
from posit.connect.external.databricks import databricks_config
from posit.workbench.external.databricks import WorkbenchStrategy
from sqlalchemy import create_engine, text

# Configure Databricks connection using environment variables.
# - Workbench: WorkbenchStrategy() provides config.token directly via managed credentials.
# - Connect: DefaultCredentials() uses file-oidc (config.token is None); bearer token
#   is resolved dynamically via config.authenticate().
# - Local: databricks_cli provides config.token via the Databricks CLI profile.
config = databricks_config(
    posit_default_strategy=databricks_cli,
    posit_workbench_strategy=WorkbenchStrategy(),
    posit_connect_strategy=DefaultCredentials(),
    host=f"https://{os.environ['DATABRICKS_HOST']}",
    warehouse_id=os.environ["DATABRICKS_HTTP_PATH"],
)

bearer_token = (
    config.token
    or config.authenticate().get("Authorization", "").removeprefix("Bearer ").strip()
)

# Create SQLAlchemy engine
connection_string = (
    f"databricks://token:{bearer_token}@{os.environ['DATABRICKS_HOST']}:443"
)
engine = create_engine(
    connection_string, connect_args={"http_path": os.environ["DATABRICKS_HTTP_PATH"]}
)

# Query data
query = "SELECT * FROM samples.nyctaxi.trips LIMIT 10"
df = pd.read_sql(text(query), engine)

df

In R, the odbc package provides a helper function odbc::databricks() to allow for seamless connections to Databricks using user-based Workbench managed credentials and Connect workload identity OAuth credentials without needing to adjust your code. This is the recommended approach for connecting to Databricks SQL warehouses from R content being developed on Workbench and published on Connect.

When running on Connect, odbc (>= 1.7.0) automatically detects the workload identity credentials from the Databricks environment variables that Connect injects, so the code is identical to the Posit Workbench Managed Credentials section. See the databricks() helper function in the odbc package documentation for more information.

library(dplyr)
library(dbplyr)

# Workload identity federation is handled by odbc (>= 1.7.0) using the
# Databricks environment variables that Connect injects. No additional packages
# or authentication code are required.
con <- DBI::dbConnect(
    odbc::databricks(),
    httpPath = Sys.getenv("DATABRICKS_HTTP_PATH")
)

# Use dplyr to query data at "my_catalog.my_schema.my_table"
data <- tbl(con, DBI::Id("samples", "nyctaxi", "trips"))
head(data)

# Use Collect to bring the first 100 rows in memory
results <- data |> head(100) |> collect()
results

For more in-depth information and complete examples, see:

The Connect User Guide OAuth Integrations section for more detail on these concepts.
The Connect Cookbook for complete examples of using Databricks integrations in Connect.