Skip to main content
The lakehouse connector, compute-time billed over Unity Catalog. Namespace: catalog.schema.table (three-level). Cost paradigm: warehouse-seconds (DBUs shown alongside). Read-only against data, enforced in depth.

​
Authentication: discover, don’t ask

The engine discovers a connection at runtime through the Databricks SDK’s unified auth chain and never prompts for or persists a token. Discovery order:
  1. databricks.profile in .dex/config.yml, naming a ~/.databrickscfg entry (the store databricks auth login writes)
  2. DATABRICKS_* environment variables (DATABRICKS_HOST plus a credential: DATABRICKS_TOKEN, OAuth M2M client variables, or the GitHub OIDC federation CI exchanges into a token)
  3. the default ~/.databrickscfg profile (a [DEFAULT] section, or the newer CLI’s default_profile pointer)
  4. the host of a type: databricks target in a discovered dbt profiles.yml
The CLI’s OAuth cache, PATs, OAuth M2M, and workload identity federation all arrive through the same SDK door; only the coarse method (for example default_profile:oauth_user) is ever surfaced. Identities and token values never cross the envelope. Every discovery failure names the fix.

​
Config

The warehouse pin is strict. Billed statements run only on databricks.warehouse (a SQL warehouse ID or its /sql/1.0/warehouses/... HTTP path); an unpinned or unresolvable warehouse refuses with the fix named. Pin the smallest warehouse that works (a 2X-Small serverless with the minimum auto-stop is the intended shape; scripts/setup_databricks_ci.sh provisions exactly that). transform init content-checks every schema the new project would build into through Unity Catalog REST only, so the pinned warehouse is never woken: the base dev_catalog.dev_schema, plus the sibling layer schemas (staging_dev, intermediate_dev, marts_dev on the dev target) when --layered-schemas is on. A schema that already holds tables or views is warned about, never refused.

​
Cost model: two clients, one guarded quantity

Metadata is free and scans cost warehouse runtime, so dex guards warehouse-seconds, not bytes, and splits its clients to keep the free paths provably free:
  • Free (SDK REST, no SQL session): connect test, explore inventory, all catalog/schema/table/column metadata, warehouse facts, estimation, and maintain check (drift detection reads metadata only). The SQL session is opened lazily by the first billed statement, because opening one lands on the warehouse and can wake it; free commands leave a STOPPED warehouse stopped.
  • Billed (SQL session on the pinned warehouse): profiling aggregates, explore query, relationship verification probes, DESCRIBE DETAIL size probes, and transform build.
explore query and explore cluster bill an auto-profile of an object this connection has that the .dex/ cache cannot adjudicate, priced into the same handshake as the statements: see �3. Pass --no-auto-profile (or set auto_profile: false in .dex/config.yml) to be refused instead. Budgets (budget.ceiling, --budget, budget.session_ceiling) are warehouse-seconds: the number you budget is the number the server enforces. Every cost surface also carries the translation to DBUs (seconds x dbu_per_hour / 3600, rate from the warehouse size, marked approximate: published rates move and differ by type and cloud), and to dollars when databricks.dbu_price_usd is set. dex never guesses a dollar price: it varies by cloud, tier, and contract. Estimates start as an honest floor and refine in-budget. Databricks has no dry-run and Unity Catalog exposes no free row or byte counts, so the first handshake quotes a conservative per-statement floor, labeled estimate_quality: "low", plus a startup floor when the pinned warehouse is not running (seconds for serverless, minutes for classic). Once you confirm a budget, the adapter runs an engine-built DESCRIBE DETAIL per table, charged inside the confirmed budget, never before it, to learn sizeInBytes: that sharpens later estimates, drives the sampling decision, and is noted per table when unavailable (some shared tables refuse it). The same estimator prices transform build: each compiled model, snapshot, and test is estimated and summed into the build’s upfront cost, labeled estimate_quality: "low" like any other Databricks estimate. transform build --verify costs only where the warehouse keeps no row count, so a table’s count is free metadata; how the counts are priced into the build’s own estimate is in �1. transform test --mutate prices its whole batch as one number and confirms it once, then runs one dbt invocation per mutant. Nothing is materialized: a mutant builds as an ephemeral model, so the dev namespace holds exactly the relations it held before. A budget that runs out partway stops the run, and the remaining mutants are reported not_run rather than the budget being exceeded. Survivors are not yet compared with the model on the dev data on this connector, so each reports equivalence.status: not_checked with the reason, and nothing is priced or run for it. --verify also reads the dev schema (databricks.dev_catalog / databricks.dev_schema) for the length of that one command, because dbt writes the relations it is judging there and that schema is refused as a source everywhere else. The schema need not exist yet: dbt creates it on a first build, and dex lists it after dbt has run. How that read is bounded is under Read scope in �3. The budget is hard-enforced regardless of estimate quality. Before every billed statement the session’s STATEMENT_TIMEOUT is set to the remaining budget, so a weak floor cannot overrun the ceiling: Databricks kills the statement and dex reports the over-ceiling refusal. Actual spend is wall-clock seconds per statement (including any wake the statement caused), recorded as billed_seconds in the ledger. Connections identify themselves with a dex user-agent for attribution. The estimate carries its DBU translation; the handshake itself is in �0.

​
Read-only, enforced in depth

  • Every data statement passes the SELECT-only guard parsed in the databricks dialect through one execution door (DML, DDL, MERGE, COPY INTO, OPTIMIZE, VACUUM, and multi-statement forms all refuse).
  • The adapter issues no mutating calls: Unity Catalog REST reads, SET STATEMENT_TIMEOUT, DESCRIBE DETAIL, and guarded SELECTs only.
  • The documented grant shape is least-privilege: CAN USE on the pinned warehouse, USE CATALOG + USE SCHEMA + SELECT on source catalogs, and write only on the dedicated dev catalog (scripts/setup_databricks_ci.sh creates this shape).
  • dbt dev builds write to dev_catalog.dev_schema, which is refused as a source, and the generated profile pins the warehouse’s HTTP path and one thread. A user OAuth connection renders dbt’s own OAuth flow; a token connection renders a DATABRICKS_TOKEN env reference, never a value.

​
Profiling behavior

Profiling is batched aggregates (COUNT, APPROX_COUNT_DISTINCT, min/max only for engine-cleared safe columns), never row values. Nested and semi-structured columns (ARRAY, MAP, STRUCT, VARIANT, BINARY, GEOGRAPHY, GEOMETRY) degrade to non-null counts. Tables above databricks.max_full_profile_bytes are profiled from a sample (TABLESAMPLE), noted, with uniqueness not judged; the threshold binds once DESCRIBE DETAIL has learned a size. Exact distinct-count escalation spends only inside the confirmed budget and degrades to approximate verdicts with a note when the remainder cannot cover it.

​
Testing

Offline: a stateful fake pair (tests/fakes/databricks.py): a Unity Catalog workspace client serving metadata and a DBAPI connection that records statement ordering, simulates timing on an injected clock, and enforces STATEMENT_TIMEOUT with the connector’s real error types, behind a counting factory that proves free paths never open a SQL session. Live: an env-gated integration suite (DEX_TEST_DATABRICKS_*) reading the samples catalog and writing only to the scratch catalog, run in CI on a schedule via an OIDC federation policy (no stored keys), never as a merge or release gate. Source: references/databricks.md (edit on main). This page is generated; change the source, not the site.