Data Agent Service
v0.2.0 both executors, every SQL source →Natural-language questions over the warehouses, databases and APIs you already have — Fabric and Azure SQL, PostgreSQL, Databricks, and any REST service including a retrieval one — grounded in the glossary, metrics and schema held in OpenMetadata, fronted by Azure API Management. Each query is authorized twice: role rules in the service, then the source itself under the caller's own identity.
It runs locally on an emulator family, and against real Azure by changing configuration only.
Every capability below carries the command that proves it. Where something is designed but not built, this page says so.
A language model that writes SQL is a demo. A language model you let near the revenue number is an engineering problem, and it fails in three specific ways.
Contoso's fiscal year starts 1 April, so July trading is Q2, not Q3. A model reading column names alone gets a confident, wrong number — and nothing in the output says so.
Meaning lives in OpenMetadata, retrieved per questionMost agents query with the service's credentials. Every user then sees everything the service can see, and the warehouse's own access control stops applying.
On-behalf-of: the engine sees the caller, not the appWithout an accuracy suite and an audit trail, "it seems to work" is the whole quality argument — and it stays that way until someone is misled by it.
Scored evals, plus an audit line per queryStandard protocols end to end — MCP, OAuth 2.0 on-behalf-of, managed identity, TDS. No component knows whether it is talking to an emulator or to Azure.
— capabilities, each with the evidence behind it and an honest status.
Four adapters behind one interface: TDS (Fabric, Azure SQL, Synapse), PostgreSQL,
Databricks, HTTP. A new source is a config entry, not a fork — and the dialect-aware
guard emits TOP or LIMIT without anyone choosing.
The OpenAPI document is the allow-list: unsafe operations are never offered, an undeclared parameter is refused rather than dropped, the ceiling is written into the request. No parse tree here — every property was translated, not ported.
10 witnesses · OpenMetadata's own API is the live sourceA knowledge base plugs in as a REST source. The usual hole in enterprise retrieval: the vector store is queried as a service, so a passage the asker may not read reaches their answer and nothing records it. Here the source declares whose permissions it applies, and denied fields are stripped as warehouse columns are.
top_k capped by the deployment, not the model · docs/15-http-sourcesA standards-compliant MCP server over Streamable HTTP, no vendor extensions. Proved by two independent official SDKs — Python and TypeScript — so a server only its own language can drive fails rather than passes.
28 client checks · config generated for Claude, Cursor, VS CodeThe user's token is exchanged (OAuth 2.0 OBO), so the engine applies their permissions, not the service's. Table and column rules run on the SQL parse tree, and a withheld column disappears from schema discovery too.
3 personas: same question, different answers, one denialk6 drives four scenarios: gateway, executor direct, catalog, rate limiter. Thresholds gate the run and payloads are inspected — an HTTP-only check would score a refusal as a success.
261 req/s · p95 69.5ms · gateway costs +1.6ms and ~2%31 questions across five tiers and two datasets, scored on result sets rather than SQL text. Reference answers score 100%, which calibrates the instrument. The live model has not been scored — that run needs an API key.
make eval · L3 questions fail without the catalogEvery operation writes a structured line — who asked, which source, the verdict, the columns read, and whether it ran as the user or as the service, so a weaker authorization tier is visible rather than assumed.
op · user · source · verdict · authz_tier · msRecurring ad-hoc questions are the signal a dashboard is justified. Queries are canonicalised — literals and aliases normalised away — so the same question asked three ways lands on one template, and frequency becomes evidence.
promoter/ · — witnesses · no natural language is storedOne Plan — tables, measures, dimensions, slots — spelled by each target in its
own language: a Power BI semantic model, an Apache Superset dataset and chart,
a Tableau workbook. Each is then asked the question through its own engine
and checked against the SQL the template came from. A dashboard whose two answers disagree
never reaches the catalog.
Each carries the guarded template, not the tables behind it, so no target can widen the surface the access rules narrowed.
— + — + — witnesses · identity carried where the tool can hold it, and named honestly where it cannotTableau has no container, so the line is drawn where the evidence stops. The workbook,
the VizQL query and the connected-app token are pure functions of the Plan,
witnessed on every push. Publishing to a real site is not — the
ledger carries that red rather than letting a green suite imply otherwise.
publish() refuses by name
The evals are built so that ignoring the glossary changes the answer, not just the number — because a wrong number that looks plausible is the failure mode that matters.
| Question | Naive reading | With the catalog |
|---|---|---|
| Q2 revenue | Calendar Jul–Sep | Fiscal Q2 — the year starts 1 April |
| Revenue | SUM(amount_usd), cancellations included |
Net of cancellations — about 5% of web orders |
| Fastest support team | Elapsed time: Billing is last at 502 min | Resolution time excludes customer wait: Billing is first at 210 min |
The naive answer does not merely round differently — it names the wrong winner. A ranking flip cannot be explained away, so the eval fails loudly instead of drifting quietly.
This project's discipline is that a claim is worth what checks it. The same rule applies to this page.
| Capability | Status | What backs it |
|---|---|---|
| Fabric + PostgreSQL, both executors | shipped | 28/28 contract against each; two datasets live in two engines |
| Databricks adapter | shipped, unwitnessed | In executor-py with unit tests, over the Statement Execution API — but never run against a real workspace, and absent from the Go executor |
| Fabric Lakehouse | untested | Its SQL analytics endpoint is the same TDS surface the Warehouse exposes, so the existing adapter should reach it — nothing here has tried |
| Snowflake | not built | ~150 lines against External OAuth trusting Entra; plan §15 |
| DuckDB | not supported | No adapter and no dialect entry — it appears nowhere in the code |
| MCP, client-agnostic | witnessed | 28 checks; two independent official SDKs |
| OBO per-user authorization | witnessed | Three personas, including a denial |
| Guardrails on the parse tree | shipped | Unit corpus shared by both implementations |
| Load testing | measured | Four k6 scenarios with gating thresholds |
| Eval harness and gold baseline | shipped | 31 questions, five tiers, result-set scoring |
| Live model accuracy score | measured, small n | Through Claude Code, not the SDK. Catalog vs none: pass 71% vs 50%, semantics 100% vs 40%, L3 +60 points. That run covered 14 support questions on one repeat — directional, not statistically strong, and predates the L3 expansion |
| Audit trail | shipped | Structured line per operation, authz tier recorded |
| Production parity (real Azure) | gated, unwitnessed | Terraform validates, runbook and preflight exist; not yet run against a tenant |
| Dashboard promotion | shipped | Templates counted by pseudonymous user, k-threshold, catalog-derived titles; no question stored |
| Power BI target | shipped | Direct Lake semantic model and PBIR report, published under OBO as the asking person, verified by evaluating its own DAX against the SQL; rendering prod-only |
| Apache Superset target | shipped | Virtual dataset carrying the guarded template, chart and dashboard; verified through Superset’s own query layer. authz_tier: service — Superset has no on-behalf-of exchange, so the asker is recorded as an OpenMetadata owner and the docs say so rather than implying otherwise |
| Tableau target | generator shipped, hop unwitnessed | Workbook, VizQL Data Service query and connected-app token are pure functions of the Plan, recorded in the contract and witnessed in CI. authz_tier: user via direct trust. Publishing to a real site needs a tenant — red in the ledger until one is witnessed |
| A fourth dashboard target | not built | Cube, Looker, Metabase and the rest are a DashboardTarget each — see adding a dashboard target. The filters that decide whether one is worth building are written down there |
| Existing MCP server as a source | shipped | APIM mcpMode: passthrough; OpenMetadata's own MCP server reaches the agent this way in every run |
| REST source, no adapter (APIM REST→MCP) | blocked upstream | The pinned gateway builds a fresh request and copies no inbound headers, so on-behalf-of sees an anonymous caller (upstream #8) |
| REST source adapter (incl. retrieval APIs) | shipped | Python executor; guarded against the OpenAPI document, registered in OpenMetadata as an apiService, 10 witnesses |
| GraphQL source adapter | not built | Needs a query parser with depth and cost budgets — closer to the SQL guard than to REST |
| Operation-level eval metrics | not built | Result-set comparison is meaningless for retrieved prose; a KB source needs its own scoring |
Two executor images, same contract, same version. Pick either — nothing above the executor changes, and both now serve every configured source.
docker pull ghcr.io/calvinchengx/data-agent-service/executor-py:0.2.0
docker pull ghcr.io/calvinchengx/data-agent-service/executor-go:0.2.0| Image | Size | Platforms | Engines it serves |
|---|---|---|---|
executor-go | 19.5 MB | linux/amd64 · linux/arm64 | Fabric (T-SQL over TDS), PostgreSQL |
executor-py | 290 MB | linux/amd64 · linux/arm64 | Fabric, PostgreSQL, Databricks |
Choose with DAS_EXECUTOR=py|go. The Go image is
a static binary on distroless — no ODBC driver, no Kerberos libraries — which is most of the
difference in size. Why both exist, and what the comparison measured, is in
ADR 0001.
The REST adapter landed after v0.1.0 and is Python-only — the Go executor has neither it nor
a Databricks adapter, recorded in ADR 0001 rather than discovered later. GraphQL is not
built. An unknown source kind is refused outright rather than guessed at, so a
source no adapter handles fails loudly instead of being answered from the wrong engine.
One command brings up the stack from nothing, seeds it, and verifies it — the same command continuous integration runs, so a fresh clone is a tested path rather than a hope.
git clone https://github.com/calvinchengx/data-agent-service
cd data-agent-service
cp .env.example .env
make stack # up, seed, apply, verify
make ask Q="What was net revenue in fiscal Q2?"Local emulators and real Azure differ by configuration only. A gate greps the tree for endpoints written into code, target branching and development-only surfaces, so "it works the same in production" stays checkable.
make lint # ruff, ty, golangci-lint
make test # unit tests + end-to-end witnesses
make eval # accuracy, per use case
make load # k6 scenarios with gating thresholds
ENV=prod make test # the same checks against a real tenant