Databricks connection patterns on Connect using service account OAuth authentication

These access patterns enable you to connect to and read from Databricks databases or schemas in Posit Connect using Connect Service Account OAuth Authentication. They include both Python and R code examples for connecting to Databricks using Spark and SQL.

Choose a different pattern by referring to the table in Choose a connection for Databricks.

NoteNot sure which pattern to use?

For an overview of credential types and scope in different environments, see Credential types and Managing data access from development to deployment.

Overview

ImportantAdministrator configuration required

Your Connect Administrator must configure a service account type Databricks OAuth integration in Connect. Your Databricks Workspace Administrator must also create a service principal and grant it appropriate permissions to access Databricks clusters and SQL warehouses. For detailed setup instructions, see the Databricks OAuth Integrations page in the Posit Connect Admin Guide.

Service account integrations allow publishers to acquire and use the credentials of a service account (what Databricks calls a service principal) when accessing protected resources. Every viewer will have the same experience when interacting with the content.

Service account integrations support both interactive and rendered content types. Service account integrations also support access patterns using Anyone - no login required and Share links.

Required steps when publishing

You must enable the OAuth integration for your specified content using the Access tab in the Content Settings pane in Connect, or by selecting the integration upon deployment.

For more details, refer to the Adding Integrations section of the Connect User Guide.

Spark cluster connection

Python

To authenticate with Databricks using service account credentials, your code must retrieve the content session token from the CONNECT_CONTENT_SESSION_TOKEN environment variable. Connect automatically provides this environment variable when the OAuth integration is enabled. Your code must then exchange it for a Databricks OAuth access token. The posit.connect.external.databricks module from the posit-sdk package simplifies this token exchange process, with ConnectStrategy() handling the exchange when deployed to Connect.

This code example uses different authentication strategies depending on the environment:

• ConnectStrategy retrieves and exchanges the content session token when deployed to Connect
• WorkbenchStrategy uses Workbench managed credentials in Workbench
• databricks_cli provides a fallback for local development

This approach allows the same code to run seamlessly across all three environments.

import os

# ...
import pandas as pd
from databricks.connect import DatabricksSession
from databricks.sdk.core import databricks_cli
from posit.connect.external.databricks import ConnectStrategy, databricks_config
from posit.workbench.external.databricks import WorkbenchStrategy

# ...

# Configure Databricks connection using environment variables
# the `ConnectStrategy()` defaults to using the environment variable CONNECT_CONTENT_SESSION_TOKEN.
# When using a Connect service account OAuth integration, this environment variable is automatically
# set so you do not need to specify this in order to conduct the credential exchange to retrieve the
# access token.
config = databricks_config(
    posit_default_strategy=databricks_cli,
    posit_workbench_strategy=WorkbenchStrategy(),
    posit_connect_strategy=ConnectStrategy(),
    host=f"https://{os.environ['DATABRICKS_HOST']}",
    cluster_id=os.environ["DATABRICKS_CLUSTER_ID"],
)

# Create Databricks Spark session
spark = DatabricksSession.builder.sdkConfig(config).getOrCreate()

# Read a table located at "samples.nyctaxi.trips"
df = spark.read.table("samples.nyctaxi.trips")
df.show(5)

# Convert the Spark table to a Pandas DataFrame
pandas_df = df.limit(5).toPandas()

pandas_df

R

To authenticate with Databricks using service account credentials, your code must retrieve the content session token from the CONNECT_CONTENT_SESSION_TOKEN environment variable. Connect automatically provides this environment variable when the OAuth integration is enabled. Your code must then exchange it for a Databricks OAuth access token. The connectcreds package simplifies this process, with the connect_service_account_token() function handling the token retrieval and exchange when deployed to Connect.

This code example uses conditional logic to authenticate depending on the environment:

• connect_service_account_token() retrieves and exchanges the content session token when deployed to Connect
• In development environments, you can use an alternative authentication method

This approach allows the same code to run across different environments.

# ...
library(sparklyr)
library(pysparklyr)
library(connectcreds)
library(dplyr)
library(dbplyr)

# if running on Connect
if (Sys.getenv("POSIT_PRODUCT") == "CONNECT") {
  sc <- spark_connect(
    cluster_id = "<your-cluster-id>",
    method = "databricks_connect",
    token = connectcreds::connect_service_account_token()$access_token
  )

  # if running outside of Connect (in development)
} else {
  access_token <- #<utilize your desired method for retrieving the access code used in development>
  sc <- #<make spark connection to Databricks using example access patterns in this document>
}

# Interact with the Spark table at "my_catalog.my_schema.my_table" as a dplyr data frame
data <- tbl(sc, dbplyr::in_catalog("samples", "nyctaxi", "trips"))
head(data)

# Use Collect to bring the first 100 rows in memory
results <- data |> head(100) |> collect()
results

For more in-depth information and complete examples, see:

• The Connect User Guide OAuth Integrations section for more detail on these concepts.
• The Connect Cookbook for complete examples of using Databricks Service Account Integrations in Connect and a fallback approach for local development.

SQL warehouse connection

Python

To authenticate with Databricks using service account credentials, your code must retrieve the content session token from the CONNECT_CONTENT_SESSION_TOKEN environment variable. Connect automatically provides this environment variable when the OAuth integration is enabled. Your code must then exchange it for a Databricks OAuth access token. The posit.connect.external.databricks module from the posit-sdk package simplifies this token exchange process, with ConnectStrategy() handling the exchange when deployed to Connect.

This code example uses different authentication strategies depending on the environment:

• ConnectStrategy retrieves and exchanges the content session token when deployed to Connect
• WorkbenchStrategy uses Workbench managed credentials in Workbench
• databricks_cli provides a fallback for local development

This approach allows the same code to run seamlessly across all three environments.

import os

# ...
import pandas as pd
from databricks.sdk.core import (
    databricks_cli,  # this can be removed if not using `posit_default_strategy`
)
from posit.connect.external.databricks import ConnectStrategy, databricks_config
from posit.workbench.external.databricks import WorkbenchStrategy
from sqlalchemy import create_engine, text

# ...

# Configure Databricks connection using environment variables
# the `ConnectStrategy()` defaults to using the environment variable CONNECT_CONTENT_SESSION_TOKEN.
# When using a Connect service account OAuth integration, this environment variable is automatically
# set so you do not need to specify this in order to conduct the credential exchange to retrieve the
# access token.
config = databricks_config(
    posit_default_strategy=databricks_cli,
    posit_workbench_strategy=WorkbenchStrategy(),
    posit_connect_strategy=ConnectStrategy(),
    host=f"https://{os.environ['DATABRICKS_HOST']}",
    warehouse_id=os.environ["DATABRICKS_HTTP_PATH"],
)

# Create SQLAlchemy engine
connection_string = (
    f"databricks://token:{config.token}@{os.environ['DATABRICKS_HOST']}:443"
)
engine = create_engine(
    connection_string, connect_args={"http_path": os.environ["DATABRICKS_HTTP_PATH"]}
)

# Query data
query = "SELECT * FROM samples.nyctaxi.trips LIMIT 10"
df = pd.read_sql(text(query), engine)

df

R

The odbc package provides a helper function odbc::databricks() to allow seamless connections to Databricks using user-based Workbench managed credentials and Connect viewer OAuth credentials without needing to adjust your code.

This is the recommended approach for connecting to Databricks SQL warehouses from R content being developed on Workbench and published on Connect. The only difference between this code and what is shown in the Posit Workbench Managed Credentials section is the addition of the connectcreds package, which automatically detects the environment and retrieves the viewer’s credentials when running on Connect.

See the databricks() helper function in the odbc package documentation for more information.

Note

odbc version 1.7.0 or above is required.

library(dplyr)
library(dbplyr)
library(connectcreds) # this is required for the code to run using service account credentials on Posit Connect

con <- DBI::dbConnect(
  odbc::databricks(),
  httpPath = Sys.getenv("DATABRICKS_HTTP_PATH")
)

# Use dplyr to query data at "my_catalog.my_schema.my_table"
data <- tbl(con, DBI::Id("samples", "nyctaxi", "trips"))
head(data)

# Use Collect to bring the first 100 rows in memory
results <- data |> head(100) |> collect()
results

For more in-depth information and complete examples, see:

• The Connect User Guide OAuth Integrations section for more detail on these concepts.
• The Connect Cookbook for complete examples of using Databricks Service Account Integrations in Connect and a fallback approach for local development.