Databricks connection patterns on Connect using M2M OAuth authentication

These access patterns enable you to connect to and read from Databricks databases or schemas in Posit Connect using Machine-to-Machine (M2M) OAuth Authentication. They include both Python and R code examples for connecting to Databricks using Spark and SQL.

Choose a different pattern by referring to the table in Choose a connection for Databricks.

Not sure which pattern to use?

For an overview of credential types and scope in different environments, see Credential types and Managing data access from development to deployment.

Overview

Administrator configuration required

Your Databricks workspace administrator must create a service principal and generate secrets for your use. Request the client ID and client secret from your Databricks workspace administrator to configure this authentication approach.

The administrator must also grant the service principal appropriate permissions to access Databricks clusters and SQL warehouses. For more details, see Authenticate access to Databricks with a service principal using OAuth (OAuth M2M) and the Databricks Data Source admin guide.

M2M OAuth authentication provides secure access to Databricks using OAuth tokens and service principals. Service principals are established at the workspace level and, unlike user credentials, are appropriate for one-to-many authentication scenarios. This method offers enhanced security compared to long-lived Personal Access Tokens (PATs) and works in both development and deployment environments.

Define environment variables for the service principal credentials using an .env or .Renviron file, for example, and refer to them in your code. Never hard-code credentials directly in your code.

You must include the following environment variables for authentication:

DATABRICKS_HOST
DATABRICKS_CLIENT_ID
DATABRICKS_CLIENT_SECRET

You will also need to provide:

For Spark connections, the cluster ID
For SQL warehouse connections, the HTTP path

You can store these as environment variables such as DATABRICKS_CLUSTER_ID or DATABRICKS_HTTP_PATH, or explicitly specify them in your code.

Environment variable management

When publishing content to Connect that uses M2M authentication:

Do not deploy your .env or .Renviron file with your content.
Supply the DATABRICKS_HOST, DATABRICKS_CLIENT_ID, and DATABRICKS_CLIENT_SECRET environment variables in the content settings Advanced tab.

Spark cluster connection

This connection method will be supported in a future release of sparklyr.

SQL warehouse connection

SQL/ODBC connections using M2M OAuth authentication from Connect follow the same pattern used for Workbench for Connecting to a SQL warehouse. If you are also using this credential type in Workbench, you do not need to make any changes to your code for it to connect in the same manner on Connect.

Important

For secure storage of the DATABRICKS_HOST, DATABRICKS_CLIENT_ID, and DATABRICKS_CLIENT_SECRET environment variables, do not deploy your .env or .Renviron file with your content. Rather, supply these in the content settings Advanced tab.