Adding a source

A data source is configuration. Adding one that speaks an engine already supported is an entry in DAS_SOURCES and a catalog seed; adding a new engine is one adapter behind SourceBackend, and nothing above the executor changes — not the gateway, not the agent, not the evals.

This page exists because two details cost real time and are invisible in a diff, and because the second source found two defects within minutes of existing, which is the argument for having one.

Adding a source on an engine that already works

{"name": "contoso_support", "kind": "postgres", "dialect": "postgres",
 "authz_tier": "service", "om_service_fqn": "postgres_support",
 "dsn": "postgresql://…", "schemas": ["support"]}

Then seed its catalog entry (python -m seed.govern --dataset <name>) so the agent can find it: an ungoverned source is queryable and unfindable, which in practice means unused.

DAS_DEFAULT_SOURCE decides what an unqualified tool call means. With more than one source, say which — the same table name can exist in both.

Adding an engine

Implement SourceBackend (list_tables, describe, run) in services/warehouse-query-py/sources.py, register it in BACKENDS, and add the dialect to the guard’s policy. Then make it satisfy services/conformance/run.py, which both executors already do — a new engine that cannot pass the contract is not finished.

The two things that cost time

1. Every engine wants its own delegated scope. On-behalf-of asks Entra for a token for a specific resource. Azure SQL wants https://database.windows.net/user_impersonation; Databricks wants 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/user_impersonation. A single global scope works exactly until the second engine, and the failure lands at sign-in, so it reads as an outage rather than as a misconfiguration. Hence Source.scope, defaulting to DAS_SQL_SCOPE.

2. PostgreSQL takes the Entra access token as the PASSWORD. Azure Database for PostgreSQL has no token attribute in its wire protocol: you pass the token where the password goes. That is why PostgresBackend._connect has a different shape from the TDS one, and it is not a shortcut.

authz_tier — and why the weaker tier is allowed to look weaker

Tier	Meaning	Per-user authorization rests on
user	the engine is handed a token carrying the ASKING USER	the engine’s own permissions, plus the gateway and access rules
service	no Entra trust exists; one service credential for everyone	only the gateway’s roles and DAS_ACCESS_RULES

Fabric via TDS FedAuth, Azure Database for PostgreSQL and Databricks can all be user. A plain PostgreSQL cannot: every caller looks identical to it.

The tier is on every audit line, because otherwise you cannot answer later whether a row was ever protected by the engine or only by us.

The executor deliberately does not filter rows to make service tier look safer. It could — and then the audit trail would claim the engine authorized something it never saw, and the weaker tier would read as equivalent to the stronger one. A tier that is weaker should look weaker. The contract asserts the difference as behaviour: on a service-tier source two personas issuing the same query both succeed and both audit authz_tier=service; on a user-tier source the persona without a grant is refused by the source.

The credential a service source uses

A user source reaches its engine with the caller’s own token, exchanged on-behalf-of. A service source reaches it with one of two things, and credential decides which:

{"name": "contoso_support", "kind": "postgres", "authz_tier": "service",
 "dsn": "postgresql://das@postgres:5432/support",
 "credential": "keyvault:das-support-db-password"}

{"name": "contoso_gold", "kind": "databricks", "authz_tier": "service",
 "host": "https://…", "warehouse_id": "…",
 "credential": "keyvault:contoso-databricks-pat"}

With no credential, the source is reached with this service’s managed identity — right for an engine that federates with Entra. With one, the value is resolved from Key Vault with that same identity and handed to the engine wherever its password goes: the bearer for Databricks and for an HTTP API, the DSN password for PostgreSQL. Which engine it is never enters the decision — the adapter is given a string.

Three rules, all refused at start-up rather than at the first query:

	Why
credential + authz_tier: user	a shared credential cannot carry the caller’s permissions, and the audit line would then say it did
credential + a password in the dsn	two homes for one secret, and one of them is a settings file. Keep the credential, drop the password
a scheme that is not keyvault:	a mistyped reference sent as a bearer fails at the engine with a message about the header, not about the typo. A value with no scheme at all is a literal, which is right for a key someone pastes in

The audit line carries credential=user|stored|identity beside authz_tier. The tier says the engine never saw the caller; this says what it saw instead, and without both a reviewer can reconstruct neither.

This is what makes a published data product consumable. A Databricks warehouse wants a PAT, a Snowflake account wants a password, a plain PostgreSQL wants a password — none of them federate with Entra, and before this the only way to reach them was to write the secret into DAS_SOURCES. e2e.run quality asserts that no source’s password appears in any file in this repository.

Consuming a published data product

The emulator family publishes Contoso as a data product from several platforms — Fabric, Databricks, Snowflake. Consuming one is a DAS_SOURCES entry plus a credential, and this is the configuration that was tried against a running databricks-platform-jobs with the contoso-data-product-databricks-jobs product built into it:

{"name": "contoso_gold", "kind": "databricks", "dialect": "databricks",
 "authz_tier": "service", "om_service_fqn": "contoso-databricks",
 "host": "http://host.docker.internal:18470", "warehouse_id": "wh-1",
 "catalog": "contoso", "database": "gold", "schemas": ["gold"],
 "credential": "keyvault:contoso-databricks-pat"}

host.docker.internal because the product’s stack is a different compose project on the same host; in Azure it is the workspace URL.

The Databricks product: the credential reaches it, the rows do not come back

What that run proved. The product’s PAT, put in our Key Vault, was resolved by the executor’s own managed identity, reached the product’s workspace, and authenticated: statements executed and returned SUCCEEDED. The source appeared in list_sources alongside the Fabric warehouse and the support database. authz_tier: service is the honest tier — the workspace has no Entra trust with our tenant and cannot tell our callers apart, so per-user authorization rests on the gateway’s roles and DAS_ACCESS_RULES, and every audit line says authz_tier=service credential=stored.

What it did not prove, and why not. No rows came back. The emulator returns a successful statement’s rows as a JSON string at result.text with no manifest.schema, where the documented Statement Execution API returns result.data_array and manifest.schema.columns — so the adapter, which is written against the documented API, reads zero columns and zero rows from a response that contains [[119]]. Two smaller gaps sit behind it: information_schema.tables is a stub that returns no rows, and a two-part schema.table name is not resolved against the request’s catalog. docs/upstream-issues.md 12–14 carry the repros.

The Fabric product: both halves, witnessed

The same exercise against fabric-platform-notebook-pipelines with contoso-data-product-fabric-notebook-pipelines built into it went all the way through. One DAS_SOURCES entry, no code:

{"name": "contoso_product", "kind": "fabric", "dialect": "tsql",
 "authz_tier": "service", "om_service_fqn": "contoso-fabric",
 "tds_server": "host.docker.internal:11433",
 "database": "<the warehouse's item id>", "schemas": ["dbo"],
 "credential": "keyvault:contoso-fabric-product-token"}

database is the warehouse’s item id rather than its name, because the emulator’s TDS surface takes the workspace from the server name’s first DNS label and host.docker.internal is not a workspace (docs/upstream-issues.md 16). Addressing by id is configuration, not a code path, and it is the emulator’s own documented alternative.

The credential is a token minted by the PRODUCT’s Entra, put in our Key Vault. That is what authz_tier: service means here: the product’s tenant has no trust with ours, so it cannot tell our callers apart, and per-user authorization is the gateway’s and DAS_ACCESS_RULES’s alone.

	result
SELECT COUNT(*) FROM dbo.fct_sales	474,044 — the family’s canonical sale_lines
SELECT SUM(revenue_usd) FROM dbo.fct_revenue_summary	129,341,157.67 — its canonical revenue
describe_table dbo.dim_customer as Data.Finance	6 columns
the same as Data.Analyst	4 columns, 2 withheld
audit, every line	authz_tier=service credential=stored
Python executor and Go executor	identical answers, identical withholding

The third and fourth rows are the point worth stating plainly: our access rules govern a data product this repository did not build, because the withholding is decided here from the caller’s role and not by the source.

The catalog half worked too — search_metadata through the gateway returned contoso-fabric.contoso-analytics.warehouse.fct_revenue_summary beside our own fabric_contoso.contoso_warehouse.dbo.fct_revenue_summary. One catalog, two services, om_service_fqn telling a source which one is its own. Be aware of how those entries got there: the product’s govern step defaults OM_URL to localhost:8585 and its steps run on the host, so it catalogued itself into THIS repository’s OpenMetadata rather than its own (docs/upstream-issues.md 17). They were removed afterwards. The grounding model is right; that particular run proved it by accident.

No e2e.run witness for this either, for the same reason as Databricks: it needs a second stack, and CI has one.

The catalog half is blocked separately. The product registers its domain, its databaseService and its six metrics in its own OpenMetadata, and records contoso-databricks.contoso.gold.fct_revenue_summary in catalog.json — but creates no database, schema or table entity, so nothing resolves at that FQN and GET /api/v1/tables is empty (docs/upstream-issues.md 15). A consumer grounding on that catalog would find the service and nothing under it. Note that this is independent of the one-catalog question: om_service_fqn on a source already points at whichever service holds a product’s tables, so several products in one catalog need no code — they need the tables to be there.

Parsing result.text would make this work locally and would be a shape real Databricks never emits — the emulator-only path scripts/check_prod_paths.py --strict exists to forbid. So Databricks stays unwitnessed for the data path: the credential half is proved, the reading half waits on the emulator or on a real workspace. No witness was added to e2e.run for it, because a witness needing a second stack cannot run in CI, and a witness that cannot run witnesses nothing.

What the second source found, within minutes

Both of these were live defects that one engine could never have surfaced:

1. DAS_ACCESS_RULES named only dbo.*, so every column of the new schema was withheld from every role — including from describe_table. It read exactly like a deliberate permissions decision rather than a missing line.
2. The catalog’s group entitlement descriptions had gone stale against the rules. seed.authz regenerates them; changing DAS_ACCESS_RULES without re-running it leaves an access-certification campaign showing yesterday’s grant.

A third was found by using the source rather than adding it: the eval harness opened one Fabric connection for every use-case, which was correct only while there was one engine.

Writing the use-case that proves it

A second source proves nothing until something asks it a question that only the catalog can answer. For contoso_support that is Resolution Time, which excludes the period a ticket waited on the customer.

The measurement worth copying: don’t settle for a rule whose misuse changes the magnitude — find one whose misuse changes the answer. Billing tickets wait on customers most, so on wall-clock Billing is the slowest team (502 minutes, last of three) and on Resolution Time it is the fastest (210 minutes, first). An agent that reads column names and stops names the wrong winner, and a wrong winner cannot be explained away as rounding.