A Databricks workspace on your laptop, that refuses what it cannot compute.

The problem this solves

A Databricks workspace is not something you can stand up to test against. So the code that talks to one gets tested against a mock you wrote, which agrees with you by construction, or against a shared dev workspace that is slow, costs money, and is never in a known state.

Both failure modes are the same failure: the thing you test against is not the thing you ship against. A mock returns what you told it to. A shared workspace returns what the last person left behind.

This runs the real API surface locally, and is deliberate about the line between what it really does and what it only appears to do. That line is published as a parity ledger rather than described in a README, and every row on the green side of it names the evidence.

What it actually does

How a claim gets to be green

Every green row in the parity ledger names its evidence in witnesses.json, and CI fails if a row claims something no witness backs. The kinds are not equal, and are not presented as though they were:

All 31 claims carry a ci: witness. That was not true a few months ago, and closing the gap is what found most of the defects below.

Defects our own tests could not see

Each of these passed a green Go suite and broke a real client. They are the argument for the witness rule, and they are why the rule is enforced by CI rather than by intention.

A dbt task that could never reach its warehouse

The generated dbt profile carried the origin the client uses. Inside the compute container that address resolves to the container itself, so dbt dialled itself and hung. It passed the end-to-end gate because that gate handed the emulator one address that happened to work from both sides — the two names were the same string in the test and nowhere else.

A failing dbt test that came back as a network error

When dbt failed, the generated code raised SystemExit. The statement agent answers that by closing the connection without replying, so the caller got Post /statements: EOF and lost the run_results.json that had already been printed and would have said which test failed. The failure now travels as data, beside the evidence, where nothing between the agent and the caller can separate them.

Secrets that were fetched and then dropped

A task's spark_env_vars and spark_conf were sent to the agent in fields the agent does not read. The emulator really did resolve {{secrets/scope/key}}, and then put the value nowhere — and the run reported SUCCESS. The tests passed because they asserted the secret had been resolved, one level away from whether the task ever saw it.

Two concurrent tasks reading each other's parameters

Task parameters arrive by assigning sys.argv, which is one object per interpreter. Two tasks in the same wave overwrote each other and each read whichever landed last. Both reported SUCCESS. A loop writing one output per input silently produced fewer outputs than it had inputs, and the run was green.

Who drives it in CI

Six unmodified clients, pinned to exact versions in the lockfile. A floating install is not a witness — it is whatever the registry shipped the morning CI ran.

The engine that writes is never the engine that confirms: models land as Delta on disk and delta-rs — which never spoke to dbt, the agent, or Spark — reads the log and the rows back.

What it refuses to do

Twenty-six surfaces are marked not-implemented in the ledger, on purpose and by name. A refusal that says what is missing is a feature; a plausible empty response is a bug you find in production.

• Executing a JAR. spark_jar_task asks this process to run a Java main class. It says so and stops.
• Anything with a cluster lifecycle. python_wheel_task, libraries, instance pools, init scripts — the emulator does not own a cluster, so it will not pretend to configure one.
• Unity Catalog grants. An honest 501, not an empty list that reads as "no grants".
• Photon and DBR compatibility. The dialect is Spark SQL. Where that differs from Databricks, the difference is documented rather than papered over.
• SCIM, Permissions, Apps, SQL Alerts. Absent, and listed as absent.

Run it

docker run --rm -p 8443:8443 ghcr.io/calvinchengx/databricks-emulator:latest

# the admin PAT is written to the data directory on first start
export DATABRICKS_HOST=https://localhost:8443
export DATABRICKS_TOKEN="$(cat data/admin.pat)"
databricks workspace list /

It is one binary with no cgo. Jobs need an engine attached; with none, a run fails naming the missing engine and never reports SUCCESS. The quick start attaches one in a single compose file.