How the emulator’s surface maps to real Fabric (as documented at
learn.microsoft.com/fabric /
MicrosoftDocs/fabric-docs), and
— the point of this table — whether real work happens or just the API shape.
The emulator’s design bet is that the durable, testable surface is
contracts + storage + identity + orchestration, and those are done for real
(real signed JWTs, real Delta bytes on disk, real RBAC, a real pipeline
interpreter, real cross-engine SQL, real Livy high-concurrency session packing).
The heavyweight or proprietary compute engines are either bring-your-own
(Spark behind the Livy proxy — which is how Fabric itself layers a Livy endpoint
over Spark) or honestly stubbed.
“Real via our own wire-protocol implementation.” A row is 🟢 Real not
only when an external engine/client does the work, but also when the emulator
itself implements Fabric’s wire protocol and the logic behind it — so a real,
unmodified client gets byte- and behaviour-identical responses. Fabric’s
control plane, OneLake’s ADLS/Blob surfaces, the Data Pipeline expression
language + control flow, and the Livy high-concurrency session-packing layer
are all in this category: no engine is being proxied, yet the observable
contract matches real Fabric because we built the protocol, not a mock of it.
Where a row’s execution still needs a heavyweight engine (a REPL’s Spark
statements, a notebook’s cells), that part is split out as 🟠 BYO-engine or 🔴.
| Meaning |
|---|
| 🟢 Real | Genuine work: real signed JWTs, real bytes on disk, a real engine/client computes, real logic enforced — no pretending. |
| 🟡 Emulated | Faithful API contract + persisted state, but no engine — status is clock-derived / management-only. |
| 🟠 Bring-your-own-engine | Real when a real external engine is attached (Spark via the Livy proxy; notebook cells on the Spark sidecar); contract-only (honest 501) otherwise. |
| 🔴 Not implemented | Honest 501 or absent. |
| Fabric feature | Emulator | Type |
|---|
| Workspaces CRUD | Full | 🟢 Real (state persists) |
| Items CRUD + 12 typed collections | Full | 🟢 Real |
| Role assignments / workspace RBAC | Enforced from the validated bearer principal | 🟢 Real |
| Folders | Full | 🟢 Real |
| Capacities (list, assign / unassign) | Full state, no billing/SKU enforcement | 🟢 Real state |
| Long-running operations (202 → poll) | Clock-derived | 🟡 Emulated |
Item job execution (jobs/instances) | Generic items: status clock-derived. DataPipeline jobs really run the interpreter (see Data Factory) and set terminal status from the run | 🟡 Emulated / 🟢 Real (pipelines) |
| Fabric feature | Emulator | Type |
|---|
| Entra OAuth2 tokens / JWKS / client-credentials | entra-emulator mints real signed JWTs | 🟢 Real |
| Workspace managed identity handshake | Provisioned via entra admin API; the identity’s own token passes RBAC | 🟢 Real |
| Key Vault references in connections | Resolved against azure-keyvault-emulator | 🟢 Real |
| Tenant settings / audit / admin-portal APIs | — | 🔴 Not implemented |
Purview / lineage / sensitivity labels (governance/) | — | 🔴 Not implemented |
| Fabric feature | Emulator | Type |
|---|
| ADLS Gen2 DFS surface (create → append → flush, ranged read, list) | Full, incl. the x-ms-range dialect | 🟢 Real (real bytes) |
| Blob surface | Full | 🟢 Real |
| Delta commits (put-if-absent atomicity) | Real; -race-tested concurrent-commit race | 🟢 Real |
| Shortcuts (OneLake → OneLake) | Symlinks with target-side RBAC (trusted-workspace-access) | 🟢 Real |
| Shortcuts to external targets (S3 / ADLS Gen2 / Dataverse) | — | 🔴 501 |
| Fabric feature | Emulator | Type |
|---|
| Lakehouse item + Tables/Files storage | Full (via OneLake) | 🟢 Real |
| Notebook authoring / definition round-trip | Full | 🟢 Real |
notebookutils / mssparkutils (fs, credentials, getSecret, lakehouse, runtime) | Functional stdlib shim (python/notebookutils) | 🟢 Real |
| Spark session / statement / batch via the Livy API | Native termination (--spark-agent-url): the emulator implements the Livy REST contract itself and drives a Spark statement-executor agent. Interactive sessions are persistent REPLs whose PySpark statements are computed by real Spark (state survives across statements); batches run a script fetched from OneLake through the agent — all in e2e/livy. No Apache Livy server (it’s retired). Alternatively --spark-livy-url reverse-proxies an external Livy server | 🟢 Real |
| Notebook cell execution | The emulator parses the notebook into cells (real Go parser) and records/serves the run; real Spark executes the cells against OneLake and reports back, finalising the job’s status + exit value (e2e/notebook-run, real Delta lands). Cells stay “parsed, Pending” if no engine runs | 🟢 parse+run-record / 🟠 Spark exec |
| Livy High-Concurrency (5-REPL) sessions | Fabric’s own packing layer, implemented for real (not proxied): sessionTag packing into a shared session, 5-REPL cap + spill, non-idempotent acquire, independent get/delete, slot reuse on release. With --spark-agent-url, REPL statements run on real Spark — each REPL its own agent namespace, so the 5-REPL model is real end to end (e2e/livy) | 🟢 Real |
| Environments, Spark Job Definitions | Item management only | 🟡 Emulated |
The engine behind the agent is Sail (Rust
Spark-Connect, no JVM). Every row is probed in CI (e2e/sail), not
inferred — the fidelity deltas a Fabric notebook author actually hits:
| Notebook pattern | Emulator (Sail) | Type |
|---|
abfss://…@onelake.dfs.fabric.microsoft.com/… production paths | Work unmodified (endpoint override routes the Hadoop URL form) | 🟢 Real |
| Delta write/read/append; SQL over temp views | Full | 🟢 Real |
Time travel option("versionAsOf", n) | Works (SQL VERSION AS OF is a Sail gap) | 🟢 Real / 🔴 SQL form |
MERGE INTO | Works against a registered table target (CREATE TABLE … USING delta LOCATION); path-based delta.`az://…` merge targets don’t resolve | 🟢 Real (registered) / 🔴 path target |
createDataFrame(local_rows) | Works (runners preset localRelationSizeLimit) | 🟢 Real |
sc / RDD API / spark._jvm | Fidelity inversion: works on real Fabric, impossible on Spark Connect — the agent binds sc to a guide-rail stub that raises a clear pointer instead of NameError | 🔴 by architecture |
DML row-count envelopes (INSERT/MERGE counts) | Statement executes; DataFusion’s uint64 count is absorbed as an empty result by the SQL agent | 🟡 Emulated envelope |
Structured streaming, OPTIMIZE/VACUUM, CDF, Java/Scala UDFs, spark.jars | Absent in Sail v0.6.6 | 🔴 Not implemented |
| Concurrent Delta writers to one table | No conflict detection — both “succeed” where real Fabric/delta-spark would conflict-detect; single-writer flows unaffected | 🔴 divergence |
| Fabric feature | Emulator | Type |
|---|
| SQL-analytics-endpoint semantics over lakehouse Delta | DuckDB runs real SQL (aggregation / join / filter), e2e | 🟢 Real (engine in e2e) |
| Warehouse item management | Full | 🟢 Real |
| T-SQL over TDS + Entra FedAuth | Pure-Go TDS front (internal/tds) terminates the FedAuth handshake (real Entra token, database.windows.net audience), then byte-splices the client’s post-login session to a real per-item SQL Server connection so the engine emits every token itself. Unmodified go-mssqldb and Microsoft ODBC Driver 18 (pyodbc) clients connect and run T-SQL — including RPCs, prepared statements, and transactions. Verified against a real SQL Server; Microsoft’s real dbt-fabric adapter passes debug/seed/run/test end-to-end (e2e/dbt-fabric/) | 🟢 Real (front) / 🟠 SQL Server sidecar |
| Lakehouse SQL analytics endpoint — Delta → engine | The emulator reads the lakehouse’s Tables/<t> Delta in pure Go and reflects (CREATE+INSERT) it into the sidecar on connect, so SELECT hits real OneLake data (matches DuckDB), read-only (writes rejected). Not PolyBase — SQL Server reading Delta in place is a proven dead-end on the Linux container (a throwaway spike; see 16-warehouse-tds.md) | 🟢 Real (reflection) |
| Warehouse — read-write T-SQL | Client CREATE/INSERT/SELECT relay straight to the sidecar; the warehouse owns its data (no reflection) | 🟢 Real (relay) |
Fabric SQL Database (database/) — OLTP + OneLake mirror | Same read-write TDS/FedAuth path (its own SQL Server database), plus mirroring: POST …/sqlDatabases/{id}/refreshMirror snapshots every table to OneLake as Delta (real Parquet + _delta_log), so Spark / DuckDB / delta-rs query the operational data. Verified with a go-mssqldb-writes → mirror → Delta-reads-back e2e (gated). Continuous/CDC mirroring and write-back-to-Delta are the deferred edge | 🟢 Real (snapshot mirror) |
| Per-item isolation (each item = its own SQL Server database) | Lakehouse/Warehouse routed by type; per-item databases so they never collide | 🟢 Real |
| RBAC → SQL permissions | Workspace role enforced on connect: no role → rejected; Viewer → read-only; Contributor+ → read-write (warehouse) | 🟢 Real |
information_schema / sys.* introspection | Relays natively — reflected/warehouse tables are real SQL Server tables | 🟢 Real (relay) |
| Per-column type fidelity (real SQL types over the wire) | The splice forwards SQL Server’s own COLMETADATA, so every column carries its true native type over the wire (the re-encode fallback, used only by fake test backends, synthesizes INTN/FLTN/BITN and falls back to NVARCHAR text) | 🟢 Real (native) |
| Connection by item name (vs GUID) | Workspace read from the server name (<workspace>.datawarehouse.fabric.microsoft.com), item resolved by display name; a GUID still resolves by id (back-compat). Verified with a real go-mssqldb client | 🟢 Real |
| Fabric feature | Emulator | Type |
|---|
Data Pipeline control flow (If / ForEach / Until / Switch / Filter / Fail, expression language, dependsOn) | Pure-Go interpreter that really executes | 🟢 Real (orchestration) |
| Per-activity policy — retry + backoff + timeout | Applied to every activity type: policy.retry re-runs a failed activity (each retry from scratch; only the final outcome is recorded, carrying retryAttempt); policy.retryIntervalInSeconds is folded into the run’s durationInSeconds as virtual backoff; policy.timeout fails an attempt whose own virtual duration exceeds the limit. No real sleeping — backoff and timeouts are exercised in milliseconds on the controllable clock | 🟢 Real |
ForEach sequential / parallel (isSequential, batchCount) | Iterations run in array order (deterministic); the mode sets the reported wall-clock — sequential iterations add, a parallel batch costs its slowest — matching how real Fabric overlaps them | 🟢 Real |
List pagination (continuationToken) | Opt-in via ?maxPageSize on list endpoints (workspaces, items, capacities, folders, connections, role assignments, shortcuts): returns a page + a continuationToken/continuationUri when more remain; omitted → the full set | 🟢 Real |
| Invoke pipeline (ExecutePipeline) | Resolves the referenced DataPipeline (GUID or name, optional other workspace) and runs it through a fresh interpreter — real recursive interpretation, one level deeper on the same engines. waitOnCompletion (default) gates the parent on the child’s terminal status; parameters flow into the child; a cycle or excessive nesting fails loudly | 🟢 Real |
| Pipeline → notebook activity (TridentNotebook) | Resolves the notebook reference and creates a real RunNotebook job instance the pipeline gates on — the pipeline→jobs linkage is real; the notebook’s cells execute only on the Spark sidecar (otherwise the job is clock-derived, like any RunNotebook job) | 🟢 Real chain / 🟠 exec |
queryactivityruns detail | Full | 🟢 Real |
| Copy activity — OneLake → OneLake | Really moves the bytes through the storage layer: a file, or a directory subtree preserving structure; source/sink locations {workspaceId?, itemId, path} are expression-resolved (GUID or name); returns real filesWritten / dataWritten. External stores / format transformation are out of scope and fail loudly | 🟢 Real (in-family) / 🔴 external |
| Lookup activity — OneLake CSV/JSON/Parquet/Delta | Reads real rows from a CSV, JSON, or standalone Parquet file, or a lakehouse Delta table (Tables/<name>, auto-detected — no format hint needed) in OneLake; honors firstRowOnly; the result flows into @activity(…).output for downstream steps. Parquet/Delta reuse the warehouse’s own Parquet reader — a real Delta column keeps its native type (int/float/bool), not a stringified cell | 🟢 Real (CSV/JSON/Parquet/Delta) |
| GetMetadata activity — OneLake path | Stats a real OneLake path: exists / itemType / size / lastModified / childItems; a missing path honestly returns exists:false | 🟢 Real |
| Script / SqlServerStoredProcedure activities | Run real T-SQL against a Warehouse/Fabric-SQL-Database item’s own SQL Server database — the same per-item backend the TDS endpoint and the SQLDatabase mirror share. Script runs each scripts[] entry (Query → real rows back, NonQuery → rows affected); SqlServerStoredProcedure calls a real stored procedure with named parameters. The target is named directly as {workspaceId?, itemId} (the emulator’s own scoped mapping — real Fabric’s linkedService/connection reference isn’t modeled), the same shape Copy/Lookup/GetMetadata already use. Honest error without a warehouse SQL backend attached | 🟢 Real (scoped) |
| Web / external-connector leaves | Stubbed success — reached in dependsOn order and inputs resolved, but nothing executes: Web calls to arbitrary URLs would break the offline/deterministic guarantee | 🟡 Emulated |
| Dataflow Gen2 (Power Query M engine) | An in-pipeline Dataflow activity fails with an explicit “not implemented” | 🔴 Honest fail |
| Connectors / on-prem gateways | — | 🔴 Not implemented |
| Fabric feature | Emulator | Type |
|---|
| Git integration (connect / status / commit / update / disconnect) | Full, real state | 🟢 Real |
fabric-cicd tool publishing | The real client round-trips definitions (e2e) | 🟢 Real |
| Deployment pipelines | — | 🔴 Not implemented |
| Fabric area | Emulator | Type |
|---|
Real-Time Intelligence — Eventhouse / KQL DB / Eventstream (real-time-intelligence/) | Item management only; no KQL / streaming engine | 🟡 mgmt / 🔴 exec |
Mirroring — Mirrored Database (mirroring/) | POST …/mirroredDatabases/{id}/refreshMirror mirrors an external SQL Server source (reached via a Connection with Basic credentials) to OneLake as real Delta — reusing the exact same mirror writer the Fabric SQL Database uses (warehouse.Mirror; same code, external source). Proven by a gated e2e: a table seeded directly on an external database (bypassing the emulator’s own per-item routing entirely) mirrors and reads back correctly. Snapshot-on-trigger, not continuous/CDC replication; other source engines (Snowflake, CosmosDB, on-prem via gateway) are out of scope | 🟢 Real (snapshot mirror, SQL Server sources) |
Power BI — Semantic Model query (executeQueries) | Real bounded DAX engine — EVALUATE, SUMMARIZECOLUMNS, measures, SUM/DIVIDE, relationship filter propagation — over the model.bim, conforming to the vendored Power BI OpenAPI. Proven by e2e/semantic-model (golden DAX oracle) and e2e/great-expectations (real GX validates the results). | 🟢 Real (DAX subset) |
| Power BI — Reports / rendering; full DAX; SemPy over XMLA | No report rendering; DAX beyond the fixture subset; and the native ADOMD.NET/XMLA transport SemPy uses (no CI oracle) — all deferred with cause | 🟡 mgmt / 🔴 render |
Data Science — ML models / experiments / MLflow (data-science/) | — | 🔴 Not implemented |
Fabric SQL Database (database/), Graph (graph/), Real-Time Hub, Copilot / IQ (iq/), Embed, Workload Dev Kit | — | 🔴 Not implemented |
| Capability | Purpose |
|---|
Controllable clock (/_emulator/clock) | Advance virtual time to drive LRO / job status transitions deterministically. |
Fault injection (/_emulator/faults, /_emulator/permissions) | Force failures / throttling / RBAC denials to test client resilience. |
| Svelte management portal | Dashboard, workspaces, operations, clock, and fault controls. |
Parity isn’t claimed from our own tests alone — each 🟢 surface is pinned against
the real, unmodified client a Fabric user runs, executed against the emulator
in CI (e2e/<client>/). If Microsoft’s own tool round-trips unchanged, the
contract holds better than any assertion we could write ourselves.
| Real client (pinned) | Surface exercised | Status |
|---|
fabric-cicd (Microsoft) | Control plane / CI-CD publish | 🟢 e2e/fabric-cicd |
Fabric CLI fab (Microsoft) | Control plane — SPN auth (MSAL) + workspace/item CRUD (Notebook, SemanticModel, Report, DataPipeline, Lakehouse), ls/get/api | 🟢 e2e/fabric-cli |
deltalake (delta-rs) | OneLake Delta write/read | 🟢 e2e/delta-rs |
azure-storage-file-datalake + Blob SDK | OneLake ADLS Gen2 DFS + Blob | 🟢 e2e/adls-sdk |
azcopy (Microsoft) | OneLake Blob multi-block transfer | 🟢 e2e/azcopy |
| DuckDB | Lakehouse SQL over Delta/Parquet | 🟢 e2e/duckdb |
| PySpark behind the Livy API | Spark sessions / statements | 🟢 e2e/spark, e2e/livy, e2e/notebook-run |
notebookutils | Notebook utility shim | 🟢 e2e/notebookutils |
go-mssqldb | Warehouse/Lakehouse TDS + FedAuth | 🟢 internal/server, internal/tds |
dbt-fabricspark (Microsoft) | Fabric Spark via Livy HC sessions | 🟢 e2e/dbt-fabricspark — debug→seed→run→test on real Spark |
dbt-fabric (Microsoft) | Warehouse TDS via ODBC Driver 18 | 🟢 e2e/dbt-fabric — debug→seed→run→test through the TDS splice |
The TDS surface now has two independent driver witnesses: go-mssqldb and
the Microsoft ODBC Driver 18 (via dbt-fabric). That second driver mattered —
it exposed a real gap: go-mssqldb tolerated a synthesized FedAuth login, but
ODBC Driver 18 took a compatibility path (prepared-statement RPCs +
sp_reset_connection under mandatory connection pooling) that desynced against a
re-encoding relay. The fix was to byte-splice the post-login session straight
to the real SQL Server (so it emits every token itself), which is exactly the
kind of driver-family gap a single-driver test never surfaces. dbt-fabricspark
likewise drives the high-concurrency Livy layer over its real Livy-session
protocol (method: livy, service-principal auth via entra-emulator).
The emulator targets Microsoft Fabric — the convergence/successor product —
not the earlier Azure analytics services Fabric replaced. That boundary is why
some adjacent dbt adapters and Azure surfaces are intentionally not built:
they belong to predecessor (often retired) products, and their Fabric-native
successors are what we emulate instead.
| Adjacent product / client | Why out of scope | Fabric-era equivalent (in scope) |
|---|
Azure Synapse dedicated SQL pool (dbt-synapse) | Different product: its own control plane (Synapse workspaces) and an MPP T-SQL dialect (DISTRIBUTION = HASH, clustered-columnstore / resource-class DDL) that our vanilla SQL Server sidecar rejects. dbt-synapse layers on dbt-fabric, so the shared SQL path is already covered by the dbt-fabric witness | Fabric Warehouse — 🟢 TDS relay |
Azure Data Lake Analytics — U-SQL / SCOPE (dbt-scope) | Retired service (EOL Feb 2024), proprietary batch language, no Fabric embodiment. The only overlap (Delta on a lake) is Spark/OneLake, already witnessed | Fabric Spark — 🟠 Livy |
| ADLS Gen1 | Retired (Feb 2024), superseded by Gen2 | — |
| ADLS Gen2 (standalone storage account) | Not missing — OneLake is the Gen2 endpoint: hierarchical namespace, the dfs filesystem API, onelake.dfs.fabric.microsoft.com. Fabric has no separate storage account to emulate | OneLake — 🟢 e2e/adls-sdk |
Rule of thumb: if a capability exists only in a product Fabric replaced, it’s out
of scope; its Fabric-native successor is what we build. “We already have the
TDS/SQL Server foundation” makes Synapse cheaper, not done — the remaining
delta is a whole MPP dialect plus a second control plane, for a superseded
target. So the two dbt adapters we build (dbt-fabricspark, dbt-fabric) are
exactly the two that hit live Fabric surfaces; the other two (dbt-synapse,
dbt-scope) target predecessor products outside the emulator’s remit.
Real Fabric’s own Livy endpoint is Microsoft’s implementation of the Livy REST
contract over their Spark platform — they honor the protocol, not the retired
Apache Livy server. And where Fabric adds its own layer on top of that
protocol — high-concurrency REPL packing, which a vanilla Livy server has no
concept of — the emulator implements that layer directly rather than proxying,
because there is nothing to proxy it to. That is the same stance throughout: the
protocol and control plane are the durable, real things (built, not mocked,
so real clients can’t tell the difference), and the compute engine is attached
(Spark) or deferred when proprietary/heavyweight (Dataflow Gen2’s M engine, KQL,
Power BI rendering, T-SQL/TDS). Every deferral fails loudly rather than
pretending to succeed. See 13-roadmap.md for the milestone
history and the deferred-with-cause rationale.