Acceldata
 ADOC
Search documentation… ⌘K

Last updated: Oct 09, 2026 08:22 UTC

Databricks Reliability

Connect your Databricks workspace on AWS or Azure to Acceldata Data Observability Cloud (ADOC) to monitor the freshness, schema drift, and quality of your Databricks data.

Prerequisites

Before you connect Databricks as a data source, make sure that you have the following.

Common Prerequisites

  • Databricks Workspace: A running Databricks workspace on AWS or Azure.
  • Data plane: A data plane in the running state. ADOC needs a data plane to run Data Reliability for this data source.
  • JDBC URL: The JDBC URL of the Databricks SQL warehouse or cluster that ADOC connects to. The URL includes the warehouse ID (for example, your-warehouse-id fromhttps://<instance>/sql/1.0/warehouses/your-warehouse-id).
  • Credentials: A Databricks personal access token, or a service principal with a client ID and client secret.

AWS-Specific Prerequisites

  • Create a Databricks Personal Access Token
    • In your Databricks workspace, navigate to User avatar > User Settings > Access Tokens.
    • Click Generate New Token, assign a nickname (for example, ADOC-Token), and set an expiration date.
    • Copy the token (displayed only once) and store it securely.
  • (Optional) Store credentials in a secret manager
    • Store the personal access token or the service principal client secret in a secret manager that’s configured for your data plane. For more information, see Secret Management.
    • When you add the data source, turn on Use Secret Manager and enter the secret’s key or name instead of the credential itself.

Azure-Specific Prerequisites

  • Create an Azure Service Principal
    • In Azure Portal, go to Microsoft Entra ID > App registrations > New registration.
    • Name the app (for example, ADOC-Databricks-SP), and then click Register.
    • Copy the Application (Client) ID, Directory (Tenant) ID, and create a Client Secret (under Certificates & secrets).
    • Save these values securely.
  • Create and Assign a Custom Role
    • In Azure Portal, navigate to Subscription or Databricks Resource Group > Access Control (IAM).
    • Click Add > Add custom role and define permissions:

{ "permissions": [ { "actions": [ "Microsoft.Databricks/workspaces/*/read", "Microsoft.Databricks/workspaces/clusters/read" ], "notActions": [], "dataActions": [], "notDataActions": [] } ] }

Assign the role to your service principal.

  • Add Service Principal to Databricks Workspace: In Databricks, navigate to Admin Settings > Users or Groups, add the Service Principal using its Client ID, and assign roles (for example, Contributor).
  • Grant Workspace Admin Access: In the Databricks user list, locate the Service Principal and enable the Workspace Admin toggle for deep monitoring.
  • Gather Workspace Details: Collect your Databricks workspace URL (for example, https://adb-xxxxxx.azuredatabricks.net) and workspace ID.

Prerequisite Comparison Table

Requirement

AWS

Azure

Identity / Auth

Personal Access Token

Personal Access Token or Service Principal

Key Permissions

Workspace / API Read

Custom RBAC Role, Admin Workspace Access

Secret Management

AWS Secrets Manager (optional)

Azure Key Vault (optional)

Add Databricks as a Data Source

Follow these steps to connect your Databricks workspace to ADOC for Data Reliability. Steps are identical for AWS and Azure unless specified.

Step 1: Start Setup

  • In the left navigation, select Control Center > Integrations.
  • On the Integrations page, select Add Data Source, and then select Databricks.
  • On the Data Source Details page:
    • In Data Source name, enter a name, for example, Prod-Databricks-Reliability.
    • Optionally, in Description, enter a description.
    • Turn on Data Reliability.
    • In Select Dataplane, select a data plane that’s in the running state.
  • Click Next.

Step 2: Add Connection Details

On the Databricks Connection Details page, enter the following details. Refer to Databricks Documentation for help finding IDs or URLs.

  • Cloud Provider: Select AWS or AZURE.
  • Databricks Workspace Name: Enter the name of your Databricks workspace.
  • Token: Enter the personal access token. This field isn’t shown when you turn on Use Service Principal or Use Secret Manager.
  • Token Expiry Time: Select the date when the token expires. This field isn’t shown when you turn on Use Service Principal.
  • Use Service Principal (optional): Turn on to connect with a service principal instead of a token. For Azure, create the service principal in Microsoft Entra ID. For AWS, create it in Databricks.
  • Enter Databricks URL (the URL of your Databricks workspace), Service Principal - Client Id, Service Principal - Client Secret, and, for Azure, AZURE Tenant Id.
  • Use Secret Manager (optional): Turn on to read the token or the service principal client secret from a secret manager instead of entering it. In Select Secret Manager, select the secret manager. Then enter the key or name of the stored secret in Secret Key/Name (for a token) or Secret Key/Name for Service Principal Client Secret (for a service principal). For more information, see Secret Management.
  • JDBC URL: Enter the JDBC URL of the SQL warehouse or cluster, for example, jdbc:spark://adb-1234567890.azuredatabricks.net:443/default;transportMode=http;ssl=1;httpPath=sql/protocolv1/o/0/your-warehouse-id.
  • Dataplane Engine: Select Spark or Pushdown as the engine for profiling and data quality checks.
  • Configure Global Storage within Databricks[For pushdown engine] (Pushdown only, optional): Turn on to configure global storage for the data source, and then enter the storage path in Base Path. File Format is set to PARQUET automatically.
  • Enable mTLS (optional): Turn on to use mutual TLS (mTLS) for REST API connections. Turn this on when Databricks is accessed through a proxy, such as MuleSoft, that requires client certificate authentication.

Step 3: Validate and Save Connection

  • Click Test Connection. If successful, you’ll see “Connected.” If it fails, check:
    • The personal access token or service principal credentials are invalid or expired.
    • The Databricks URL is incorrect.
    • The JDBC URL is incorrect, or the SQL warehouse isn’t running.
    • Insufficient permissions for token or Service Principal.
    • The selected secret manager or the secret key or name is incorrect, if Use Secret Manager is turned on.
  • Click Next.

Step 4: Configure Data Reliability

Configure settings for ADOC to monitor data reliability:

  • Unity Catalog Enabled: Toggle ON if using Unity Catalog to monitor metadata and assets.
  • JDBC Records Fetch Size: Set records per query (e.g., 1000) based on workspace size for optimal performance.
  • Enable Crawler Execution Schedule: Toggle ON and set a schedule (e.g., daily at 2 AM UTC) for automated reliability checks.
  • Click Submit to save. A Databricks card will appear on the ADOC Data Sources page, showing connection and crawler status.

What’s Next

With Data Reliability enabled, you can:

  • Monitor Data Quality: Use ADOC to track dataset freshness, schema drift, and anomalies in your cloud-based platform.
  • Set Policies: Define rules (e.g., missing-value checks, duplicates) for Unity Catalog or standard tables.
  • Visualize Health: Use ADOC dashboards to view table and pipeline health trends.

Glossary

  • Unity Catalog: Databricks’ governance solution for metadata and assets.
  • JDBC URL: A connection string for Databricks SQL Warehouses.
  • Personal Access Token: A secure key for Databricks API access.

Additional References