Choose a connection for Databricks

Overview

This section reviews the connection types and preferred credentials for accessing data in Databricks.

Access patterns for Databricks vary based on:

The type of compute needed, which determines whether you use Spark or ODBC/SQL to interact with your data. See Select a connection type for more details.
The credential method you use. Databricks supports a variety of access token-based credential methods, as discussed in the Authenticate access to Databricks resources documentation from Databricks. Posit has simplified some of these authentication workflows when using specific packages or Posit products. See Select a credential below for the recommended approaches when using Posit Workbench and Posit Connect.

Tip

Once you select a connection type and credential approach for both Workbench and Connect, refer to the linked examples for Python and R connection code.

Select a connection type

Your compute needs determine whether you connect with Spark or ODBC/SQL:

Spark is ideal for complex data transformations, large-scale or long-running data processing, or training machine learning models.
ODBC/SQL is ideal for submitting SQL queries to a Databricks warehouse when you do not need the full capabilities of Spark.

Select a tab below for the compute options and the libraries to use with each.

To use Spark on Databricks, you connect to a compute cluster governed by Unity Catalog. Databricks offers several compute types, most commonly serverless and all-purpose clusters. For details, see the Databricks Compute and Unity Catalog documentation.

Manage Spark connections to these compute clusters via databricks_connect, a Python library built by Databricks that lets you interact with Spark clusters from your R or Python session without installing Java components on your machine.

To make Spark connections:

Python: use databricks-connect directly. Once connected to your Spark cluster, you can manipulate Spark data using data frame libraries like polars or pandas.
R: use sparklyr and pysparklyr, which leverage databricks-connect behind the scenes. These libraries integrate with dplyr for familiar and easy manipulation of Spark data.

Tip

View, start, stop, and connect to your Spark cluster using the dedicated Databricks pane in RStudio Pro in Workbench, or the Databricks extension in VS Code and Positron.

You connect to a Databricks SQL warehouse over ODBC to run SQL queries against Unity Catalog schemas. SQL warehouses come in several types, most commonly serverless and classic. For details, see the Databricks SQL warehouse types documentation.

To make an ODBC/SQL connection:

Python: use databricks-sql-connector. Once connected to the database, you can manipulate data using data frame libraries like polars or pandas.
R: use DBI and odbc. Manipulate database tables with dplyr.

Administrator configuration required

To use odbc, ensure that the Databricks ODBC driver is installed on your Workbench and Connect Server. Your Posit administrator can follow the instructions on the Install Pro Drivers page of the documentation.

Tip

The Connections pane in RStudio, RStudio Pro, and Positron provides a convenient way to create and manage ODBC connections to Databricks schemas.

Select a credential

For more information on the choices in the tables below, see this guide’s Getting started section, including: Credential types and Managing data access from development to deployment.

The tables below summarize the recommended credential types for connecting to Databricks from Workbench and Connect, with links to code examples for both R and Python workflows.

Workbench

Workbench-managed Databricks credentials are the preferred authentication approach for accessing Databricks data from Workbench.

This method leverages a user’s existing Databricks credentials, eliminating the need to manage separate, long-lived authentication tokens such as personal access tokens (PATs).

Administrator configuration required

Administrators can refer to the Integrate Workbench with Databricks page in the Workbench Admin Guide for configuration.

Credential Scope	Credential
User	Workbench managed credential
One-to-many	OAuth2 machine-to-machine authentication (M2M)*

*Python only. R will be supported in a future release of sparklyr

Credential Scope	Credential
User	Workbench managed credential
One-to-many	OAuth2 machine-to-machine authentication (M2M)

Connect

Connect OAuth integrations and workload identity are the preferred authentication approach for accessing Databricks data from Connect. These methods enable secure authentication to Databricks from deployed content on Connect.

Administrator configuration required

Administrators can refer to the Databricks OAuth Integration page in the Connect Admin Guide for configuration.

Credential Scope	Credential
User	Connect viewer-level OAuth credentials
One-to-many	Connect service account OAuth credentials
One-to-many	Connect workload identity*
One-to-many	OAuth2 machine-to-machine authentication (M2M)*

*Python only. R will be supported in a future release of sparklyr

Credential Scope	Credential
User	Connect viewer-level OAuth credentials
One-to-many	Connect service account OAuth credentials
One-to-many	Connect workload identity
One-to-many	OAuth2 machine-to-machine authentication (M2M)

Not recommended or supported

The following methods are not supported or not recommended in Posit products for either R or Python.

Credential	Scope	Notes
OAuth user-to-machine (U2M)	User	Not supported on Workbench or Connect. This method works only from a local development IDE and requires interacting with a pop-up web browser to authenticate in real time. For local workflows, see the Databricks U2M authentication documentation.
PAT	User in development, one-to-many in deployment	Not recommended, because a PAT is less secure than OAuth. Using a PAT in deployed content effectively makes it a one-to-many credential, because the publisher’s token is used for all viewers. Use a more secure method such as viewer-level OAuth. If you must use a PAT, follow the Databricks personal access token authentication documentation.