How the emulator’s surface maps to real Fabric (as documented at
learn.microsoft.com/fabric /
MicrosoftDocs/fabric-docs), and
— the point of this table — whether real work happens or just the API shape.
The emulator’s design bet is that the durable, testable surface is
contracts + storage + identity + orchestration, and those are done for real
(real signed JWTs, real Delta bytes on disk, real RBAC, a real pipeline
interpreter, real cross-engine SQL, real Livy high-concurrency session packing).
The heavyweight or proprietary compute engines run by default: docker compose up auto-loads an override that starts Sail (Spark Connect) and a SQL
Server sidecar, so Livy sessions, notebook cells and the T-SQL warehouse do real
work out of the box. What stays 🟠 is the narrower set that needs a different
engine — the JVM overlay, or an opt-in profile — and what cannot be done
honestly at all is stubbed.
“Real via our own wire-protocol implementation.” A row is 🟢 Real not
only when an external engine/client does the work, but also when the emulator
itself implements Fabric’s wire protocol and the logic behind it — so a real,
unmodified client gets byte- and behaviour-identical responses. Fabric’s
control plane, OneLake’s ADLS/Blob surfaces, the Data Pipeline expression
language + control flow, and the Livy high-concurrency session-packing layer
are all in this category: no engine is being proxied, yet the observable
contract matches real Fabric because we built the protocol, not a mock of it.
Where a row’s execution needs an engine that is not the default — JVM-only
Spark surfaces, an opt-in KQL profile — that part is split out as 🟠 or 🔴.
| Meaning |
|---|
| 🟢 Real | Genuine work: real signed JWTs, real bytes on disk, a real engine/client computes, real logic enforced — no pretending. |
| 🟡 Emulated | Faithful API contract + persisted state, but no engine — status is clock-derived / management-only. |
| 🟠 Non-default engine | Real, but only on an engine that is not the default: the JVM Spark overlay (docker-compose.spark-jvm.yml) or an opt-in profile such as --profile rti. Not “bring your own”: docker compose up already starts Sail and the SQL Server sidecar, so the everyday Spark and T-SQL surfaces are 🟢. |
| 🔴 Not implemented | Honest 501 or absent. |
| Fabric feature | Emulator | Type |
|---|
| Workspaces CRUD | Full. Display names are unique tenant-wide — duplicates 409 WorkspaceNameAlreadyExists (uniqueness per the REST reference; fabric-docs covers workspace naming portal-side only) | 🟢 Real |
| Items CRUD + typed collections | Full. Display names are unique per (workspace, type) — duplicates 409 ItemDisplayNameAlreadyInUse; names stay reusable across types, which is why OneLake addresses items as name.Type. The type is validated against the documented ItemType enumeration (50 values, REST reference) — anything else is InvalidItemType, as real Fabric returns — and canonicalised case-insensitively so notebook and Notebook cannot become two types. 24 typed collections alias the generic surface; each collection segment is taken from its own reference page, since they are not derivable (GraphQLApis is capitalised, variableLibraries is not) | 🟢 Real |
| Role assignments / workspace RBAC | Enforced from the validated bearer principal | 🟢 Real |
| Folders | Full | 🟢 Real |
| Capacities (list, assign / unassign) | Full state, no billing/SKU enforcement | 🟢 Real state |
| Long-running operations (202 → poll) | Clock-derived | 🟡 Emulated |
Item job execution (jobs/instances) | Generic items: status clock-derived. DataPipeline jobs really run the interpreter (see Data Factory) and set terminal status from the run | 🟡 Emulated / 🟢 Real (pipelines) |
List Item Job Instances (GET …/jobs/instances) | An item’s runs, paged, newest first, with the same clock-derived status the single-instance read returns. Listing is itself an evaluation point for the scheduler below, so a run that has come due appears without touching the control surface | 🟢 Real |
Item Job Scheduler (…/items/{id}/jobs/{jobType}/schedules) | Fabric’s own per-item scheduler — distinct from the ApacheAirflowJob item, which delegates to a real Airflow sidecar. All four ScheduleConfig members: Cron (interval 1–5,270,400 min), Daily/Weekly (times[] ≤ 100, weekdays[]), Monthly (DayOfMonth or ordinal weekday, recurrence 1–12) — every documented bound enforced at write time, the 20-per-item ceiling as ScheduleExceedsLimit, and localTimeZoneId (Windows ids or IANA) honoured as real local wall time, so a daily 09:00 stays 09:00 across a DST change. Schedules really fire: each due occurrence starts a job instance down the same path a manual run takes, so a scheduled DataPipeline executes the interpreter and only invokeType: "Scheduled" tells them apart. Evaluation is driven by the controllable clock rather than a background worker — advancing time materialises exactly the occurrences that came due, and a past startDateTime triggers instantly with no special case. Boundary: catch-up is capped at 100 occurrences per evaluation (newest kept), because a controllable clock can be advanced a year against a one-minute Cron. 07-control-plane-api.md | 🟢 Real |
| Fabric feature | Emulator | Type |
|---|
| Entra OAuth2 tokens / JWKS / client-credentials | entra-emulator mints real signed JWTs | 🟢 Real |
| Workspace managed identity handshake | Provisioned via entra admin API; the identity’s own token passes RBAC | 🟢 Real |
| Key Vault references in connections | Resolved against azure-keyvault-emulator | 🟢 Real |
Governance domains (/v1/admin/domains) | Full admin surface: domain/subdomain CRUD (the hierarchy stops at two levels, as documented), workspace assignment (single-valued — re-assigning moves a workspace), nonEmptyOnly listing, and bulk Admins/Contributors role assignment. Deleting a domain cascades to its subdomains, assignments and roles. No tenant-admin gate: the emulator has no Fabric-administrator role model, so any authenticated principal may call these | 🟢 Real (mgmt); 🟡 tenant-admin gate |
Activity log / audit (GET /v1.0/myorg/admin/activityevents) | Real audit trail — the emulator records events as operations happen, using the documented audit vocabulary (CreateWorkspace, CreateArtifact/UpdateArtifact/DeleteArtifact from admin/operation-list.md; InsertDataDomainAsAdmin & co. with their DataDomainObjectId/FoldersToSetCounter properties from governance/domains-audit-schema.md). Enforces the documented request rules (single-quoted UTC bounds, same-day window) and pages with continuationToken/continuationUri until the token stops coming back. Nothing is synthesised at read time | 🟢 Real |
Tenant settings (GET /v1/admin/tenantsettings) | The documented TenantSetting object in full — settingName, title, enabled, canSpecifySecurityGroups, tenantSettingGroup, the three delegateTo* flags, enabled/excluded security groups (graphId/name), and typed properties validated against the documented TenantSettingPropertyType enum. Optional arrays are omitted, not null, as the reference’s sample does; seeded with the setting names that sample uses. POST /v1/admin/tenantsettings/{name}/update is the real update API (enabled required; response wrapped as {"tenantSettings":[…]}) | 🟢 Real |
Tenant-wide workspace admin (GET /v1/admin/workspaces) | The documented Workspace shape — note it differs from the user-facing surface: the envelope key is workspaces (not value) and the field is name (not displayName). Filters type/state/capacityId/name are enforced, with undocumented enum values returning BadRequest as the reference specifies; domainId is reported from the real domain assignment. Emulator workspaces are always Workspace/Active (no soft delete), so state=Deleted is legitimately empty. No tenant-admin gate — as with the other admin routes | 🟢 Real (mgmt); 🟡 tenant-admin gate |
Tenant-wide item admin (GET /v1/admin/items) | The documented Item shape across every workspace, with workspaceId/capacityId/type/state filters and the reference’s own error codes (InvalidItemType, InvalidItemState). Envelope key is itemEntities — a third spelling after the user-facing value and admin workspaces’ workspaces, which is why each came from its own reference page. Active is the only documented state, so that is all the emulator reports. Fields it does not model (creatorPrincipal, defaultIdentity, tags) are omitted rather than faked | 🟢 Real (mgmt); 🟡 tenant-admin gate |
Capacity tenant-setting overrides (GET /v1/admin/capacities/delegatedTenantSettingOverrides, POST …/{capacityId}/delegatedTenantSettingOverrides/{name}/update) | The documented CapacityTenantSetting (a tenant setting plus delegatedFrom and delegateToWorkspace, and without delegateToCapacity/delegateToDomain). The update body is the documented one — enabled required, no properties, no delegatedFrom — and the response wraps as {"overrides":[…]}. An override may only be created for a setting whose delegateToCapacity is true. Note there is no /v1/admin/capacities list API in Fabric: capacities are listed on the Core surface at /v1/capacities | 🟢 Real (mgmt); 🟡 tenant-admin gate |
Sensitivity labels (bulkSetLabels / bulkRemoveLabels) | The documented admin bulk APIs, reporting per-item successfulItems/failedItems rather than failing whole calls. Every change writes the documented SensitivityLabelEventData to the audit log — SensitivityLabelApplied/Changed/Removed with SensitivityLabelId, OldSensitivityLabelId, ActionSource 3 (Manual), ActionSourceDetail 5 (PublicAPI), ArtifactType 12, and a LabelEventType genuinely computed from label order (upgraded/downgraded/same-order/removed). The label taxonomy is emulator-provided — real Fabric gets labels and their order from Purview, which cannot be attached offline; GET /v1/admin/labels exposes it | 🟢 Real (label APIs + audit) / 🟡 taxonomy |
| Sensitivity labels → catalog | Labels are Purview Information Protection objects, not Atlas entities, so the Atlas-API route a Purview → OpenMetadata migration takes cannot carry them — assets, classifications, glossary and lineage cross; labels do not. The optional OpenMetadata profile exports them instead: the taxonomy becomes classification FabricSensitivity (mutuallyExclusive, since an item carries at most one label), each label a tag under it, and an item’s sensitivityLabel is applied to that item’s catalog entity. CI reads the tag back through OpenMetadata’s API, on two items with different labels, and asserts that clearing the label in Fabric clears the tag in OM (22-openmetadata.md) | 🟢 Real (stored in OM) |
Purview scanning / classification (governance/) | Purview itself is not attachable offline. What the emulator does model — labels, domains, the audit trail — reaches a catalog through the OpenMetadata profile (row above); scanning, classification rules and the Data Map API are not implemented | 🔴 Not implemented |
| Lineage (catalog graph) | Via the optional OpenMetadata profile: OneLake shortcut edges and executed pipeline Copy source→sink edges are persisted exactly and witnessed in OM’s graph API. Notebook edges come from the engine’s own report of what it read and wrote, or from the data plane observing it; Script/stored-procedure code is still not guessed. Catalog SSO can also be pointed at entra-emulator (22-openmetadata.md) | 🟢 Real (shortcuts + Copy) |
| Fabric feature | Emulator | Type |
|---|
| ADLS Gen2 DFS surface (create → append → flush, ranged read, list) | Full, incl. the x-ms-range dialect | 🟢 Real (real bytes) |
| Blob surface | Full | 🟢 Real |
| Delta commits (put-if-absent atomicity) | Real; -race-tested concurrent-commit race | 🟢 Real |
| Shortcuts (OneLake → OneLake) | Symlinks with target-side RBAC (trusted-workspace-access) | 🟢 Real |
| Shortcuts to external targets (S3 / ADLS Gen2 / Dataverse) | Real Amazon S3 / S3-compatible read-through: a Connection carrying the Access Key Id + Secret Access Key pair — as Basic credentials, because Fabric’s S3 connector uses authentication kind “Access Key” while the REST reference’s CredentialType enum has no AccessKey member and Basic is its only two-secret type — makes the emulator sign upstream requests with AWS SigV4 (internal/awssig, verified against AWS’s own published example signature). Witnessed by e2e/s3 against a real SeaweedFS server started with an identity config — the suite proves an unsigned GET is 403 and that a wrong secret is refused, so the pass cannot be vacuous; the object itself is written by boto3. ADLS Gen2 reads are witnessed too: e2e/azurite-shortcut drives a real SAS through the shortcut against Azurite, Microsoft’s own storage emulator, proving an unauthenticated GET is 403 and a tampered SAS is refused. Scope stated honestly — Azurite implements Blob/Queue/Table and not ADLS Gen2, so the endpoint is the Blob one; that witnesses the read path (a plain authenticated GET, identical on both endpoints of a real ADLS Gen2 account) but not DFS-specific behaviour, which stays unwitnessed offline. Dataverse remains an explicit 501. S3 read-only is by design — Fabric documents S3 shortcuts as read-only regardless of permissions, so a write there is refused locally and never reaches the target. ADLS Gen2 write-through is real: a file flushed under an ADLS shortcut is PUT to the storage account and the local buffer dropped (it would otherwise shadow later target changes), and deleting a file within a shortcut deletes it at the target, as documented. Witnessed by e2e/azurite-shortcut, which reads the result back with the Azure SDK straight from Azurite rather than trusting the emulator’s own 200 | 🟢 Real (S3 SigV4 + ADLS reads) / 🔴 Dataverse |
Engine-level claims below are measured, not asserted: the generated
Spark engine matrix probes each capability against both
Sail and the JVM overlay and is regenerated in CI.
| Fabric feature | Emulator | Type |
|---|
| Lakehouse item + Tables/Files storage | Full (via OneLake) | 🟢 Real |
| Notebook authoring / definition round-trip | Full | 🟢 Real |
notebookutils / mssparkutils (fs, credentials, getSecret, lakehouse, runtime) | Functional stdlib shim (python/notebookutils) | 🟢 Real |
notebookutils.notebook.runMultiple — DAG structure and ordering | Runs a DAG in dependency order, accepting a bare list or Fabric’s {activities, timeoutInSeconds, concurrency} shape. A failed activity’s dependents are reported skipped with the reason rather than run. A cycle, a dependency naming an activity outside the DAG, a duplicate activity name, or a malformed shape is refused. validateDAG exposes those same checks ahead of a run, and workspace accepts a name or an id | 🟢 Real |
notebookutils.notebook.run — exit values | Returns the child’s exit value, the exact string passed to notebookutils.notebook.exit(...), and "" when the child never called it. Applies to runMultiple’s per-activity exitVal too. Until 0.18.0 this returned the terminal job status, so a parent branching on a child’s exit value took one path here and the other on Fabric | 🟢 Real |
runMultiple — failure contract | Raises RunMultipleFailedException (importable from notebookutils.common.exceptions) when any activity did not complete, carrying every result on .result. Each value carries Fabric’s exitVal and exception keys; status/error ride alongside as emulator extras for local debugging and nothing should depend on them | 🟢 Real |
| Reference-run lakehouse rule | A referenced child notebook bound to a different default lakehouse than its parent is blocked, matching Fabric; a child that declares none inherits, a matching one runs, and useRootDefaultLakehouse in the arguments bypasses the check. The failure names the cause and the way out rather than “The job failed.” | 🟢 Real |
runMultiple — retry and timeouts | Per-activity retry/retryIntervalInSeconds (last error reported, no trailing wait after a final attempt) and the DAG-level timeoutInSeconds, defaulting to Fabric’s 12 hours. timeoutPerCellInSeconds is applied per cell, multiplied by the notebook’s real cell count | 🟢 Real |
runMultiple — concurrency | An explicit concurrency is honoured with a bounded pool per dependency level, 0 meaning unlimited; a level boundary stays a barrier and results keep their listed order regardless of completion order. The default diverges deliberately: Fabric defaults to 3× the CPU count, the emulator to sequential, because a harness comparing two runs needs the same sequence more than it needs speed | 🟡 Real, sequential by default |
runMultiple — child Spark sessions | Fabric runs children on isolated REPL instances within the parent’s Spark session, so they share its compute and its session-scoped state. The emulator gives each child its own session, so sibling activities cannot see one another’s temp views. Deliberate: the benefit is narrow and the change is large. Scoped in 39-run-multiple-parity-plan.md | 🟡 Isolated sessions |
| Spark session / statement / batch via the Livy API | Native termination (--spark-agent-url): the emulator implements the Livy contract and drives a persistent statement-executor agent. The default agent is a PySpark Connect client of Sail, not Apache Spark. An external Livy backend remains configurable with --spark-livy-url | 🟢 Real (default engine) |
| Notebook cell execution | The emulator parses and records the notebook run, resolves attached lakehouse/Environment metadata, and Sail executes cells by default. The same fixture runs on Spark 3.5 JVM and proves unqualified saveAsTable/spark.table bind to OneLake Tables/ | 🟢 Real (default engine; per-capability detail in engine-matrix.md) |
| Livy High-Concurrency (5-REPL) sessions | Fabric’s packing layer is implemented directly: sessionTag packing, 5-REPL cap + spill, independent lifecycle and slot reuse. Statements use Sail by default, so engine compatibility is limited to the Spark Connect subset | 🟢 Real (default engine) |
| Environments | Applied to the session, not just reported. A session carries an environmentId; the item’s requirements.txt, Spark properties and JAR declarations are resolved and handed to the agent, which installs the packages and applies sparkConfig before the first statement runs. Witnessed by e2e/environment (CI job environment), which imports a package the runtime image does not have — and asserts the same import fails on a session with no Environment, so a package that happened to be in the image cannot make it pass. JARs are reported as skipped: a Spark Connect session’s classpath is fixed at engine start. One Environment per agent process — a second bind of the same one is a no-op and a different one is refused with a reason, because Fabric isolates per container and a single process cannot; letting the last bind win would corrupt a dependency tree. /opt/wheels remains the fallback for consumers who model no Environment. 37-runtime-fidelity-gaps.md | 🟢 Real (packages + Spark config; JARs out of scope) |
| Spark Job Definitions | V1 definition/main/arguments/libraries parsing, attached lakehouse+Environment resolution, Pending→Completed/Failed callback lifecycle, and real Sail/JVM execution witness | 🟢 orchestration / 🟡 engine execution witnessed at the parse layer only |
The engine behind the agent is Sail (Rust
Spark-Connect, no JVM). Every row is probed in CI (e2e/sail), not
inferred — the fidelity deltas a Fabric notebook author actually hits.
A Spark 3.5 JVM image (docker/spark-runtime, Fabric Runtime 1.3’s engine
baseline) exists as a CI compatibility oracle (e2e/spark-jvm), and the
statement agent still has its classic-session path. It is also exposed as a
user-facing overlay — docker compose -f docker-compose.yml -f docker-compose.override.yml -f docker-compose.spark-jvm.yml up swaps the
statement agent onto it — so the JVM-only rows below are graded 🟠
(🟠 non-default engine): real with that overlay attached, unavailable on the default
engine. Verified live, not inferred: sc.parallelize([1,2,3,4]).map(x*2) .sum() returns 20 through the Livy agent, and spark._jvm.io.delta.tables .DeltaTable resolves.
| Notebook pattern | Emulator (Sail) | Type |
|---|
abfss://…@onelake.dfs.fabric.microsoft.com/… production paths | Work unmodified (endpoint override routes the Hadoop URL form) | 🟢 Real |
| Delta write/read/append; SQL over temp views | Full | 🟢 Real |
Time travel — DataFrame option and SQL VERSION AS OF | Both forms work on Sail. The SQL form was previously graded 🔴 on the strength of an assertion; the generated engine matrix probes it against both engines and it passes on each | 🟢 Real |
MERGE INTO | Works against a registered table target (CREATE TABLE … USING delta LOCATION); path-based delta.`az://…` merge targets don’t resolve | 🟢 Real (registered) / 🔴 path target |
createDataFrame(local_rows) | Works (runners preset localRelationSizeLimit) | 🟢 Real |
sc / RDD API / spark._jvm | Fidelity inversion: works on real Fabric, impossible on Spark Connect — the agent binds sc to a guide-rail stub that raises a clear pointer instead of NameError. Restored by the JVM overlay (classic session): sc.parallelize(…).map(…).sum() verified | 🔴 default / 🟠 JVM overlay |
DML row-count envelopes (INSERT/MERGE counts) | The engine’s row count reaches the client: DataFusion reports it as uint64, which the Arrow conversion to the Connect client rejects, and the agent used to absorb that failure as an empty envelope — “wrote 3 rows” indistinguishable from “wrote nothing”. It now recovers the count from the statement’s cached result relation, which does not re-run the statement (measured: a 3-row INSERT stays 3 rows through recovery, and the e2e asserts the count-back for exactly that reason) | 🟢 Real (recovered count) |
| Structured streaming | Partially supported on Sail v0.6.6, and the detail matters: readStream plans (its schema resolves) and the streaming query manager works, but neither is evidence that data flows: a readStream is unobservable without a sink, and the console sink writes to the server’s stdout where no client can see it — driving it for six seconds produced no output at all. Sail also reports no progress metrics (lastProgress is None), so there is nothing else to interrogate. No durable sink works, and those are the ones whose output can actually be verified. memory → No table format found for: memory; parquet/csv → cannot write streaming data to listing table; delta → unsupported extension node for streaming: DeltaWriteNode. Since a streaming job that cannot land data is not useful, this stays 🔴 for real work. The engine matrix’s streaming rows now assert that rows reach the sink — files read back, or a queryable table — rather than that a query object reported itself active, which is what previously made console look green. Full support on the JVM overlay | 🔴 durable sink / 🟠 JVM overlay |
OPTIMIZE / VACUUM (Delta maintenance) | Partly real on the default engine, via delta-rs: Sail’s planner has no such commands, so the Livy agent recognises those statements and runs them through delta-rs against the same table — verified compacting 3 files into 1 and vacuuming the superseded 3, rows intact. ZORDER/WHERE are refused, not ignored (they change what is compacted); use the JVM overlay for the full syntax. Proven against a real abfss://…onelake… table end-to-end through Livy REST (e2e/livy), with a negative control: the same statement with credentials forced off is refused, so the pass is not vacuous. delta-rs authenticates on its own account — a Connect client cannot read back the session’s bearer — minting from the same issuer with the same Storage audience, resolved per statement so a refreshed token needs no restart. One caveat remains, unchanged and by design: the executor differs from real Fabric (the emulator performs it, not Spark). See 20-lakesail-engine.md | 🟢 Real (delta-rs) |
Java/Scala UDFs, spark.jars | Out of scope for Sail by design, not a technical impossibility. Sail already embeds a foreign runtime — it links CPython via pyo3 (sail-python-udf) for Python UDFs — so a JVM could be embedded too. The real obstacle is that a Spark Java/Scala UDF is compiled against Spark’s own classes (InternalRow, Catalyst encoders), so running one needs Spark’s jars loaded — which is JVM Spark, exactly what Sail exists to avoid. Bytecode translation (GraalVM, TeaVM) does not help: the problem is Spark’s API surface, not executing bytecode. The JVM overlay is the answer | 🔴 default / 🟠 JVM overlay |
| Change Data Feed | Read is real on the default engine: spark.delta_change_feed(uri) serves a genuine change feed from OneLake through delta-rs (verified in e2e/livy, 2 versions → 2 rows). Writing a CDF-enabled table is not possible on Sail — its writer answers Unsupported table features required: [ChangeDataFeed] even with the property and the feature both named, so the table has to come from delta-rs or the JVM overlay. Exposed as a named helper rather than an interception of spark.read, so it is always obvious which engine answered | 🟢 read / 🔴 Sail-authored table / 🟠 JVM overlay |
spark.jars | Accepted but inert on Sail: JARs have no classloader. The JVM overlay has a real one (verified) | 🔴 default / 🟠 JVM overlay |
| Concurrent Delta overwrite writers | Two independent Connect sessions race at one barrier; one commits and the other receives a transaction failure from the conditional Delta-log create | 🟢 probed conflict rejection |
| Fabric feature | Emulator | Type |
|---|
| SQL-analytics-endpoint semantics over lakehouse Delta | DuckDB runs real SQL (aggregation / join / filter), e2e | 🟢 Real (engine in e2e) |
| Warehouse item management | Full | 🟢 Real |
| T-SQL over TDS + Entra FedAuth | Pure-Go TDS front (internal/tds) terminates the FedAuth handshake (real Entra token, database.windows.net audience), then byte-splices the client’s post-login session to a real per-item SQL Server connection so the engine emits every token itself. Unmodified go-mssqldb and Microsoft ODBC Driver 18 (pyodbc) clients connect and run T-SQL — including RPCs, prepared statements, and transactions. Verified against a real SQL Server; Microsoft’s real dbt-fabric adapter passes debug/seed/run/test end-to-end (e2e/dbt-fabric/) | 🟢 Real (front + default sidecar) |
| Lakehouse SQL analytics endpoint — Delta → engine | The emulator reads the lakehouse’s Tables/<t> Delta in pure Go and reflects (CREATE+INSERT) it into the sidecar on connect, so SELECT hits real OneLake data (matches DuckDB), read-only (writes rejected). Not PolyBase — SQL Server reading Delta in place is a proven dead-end on the Linux container (a throwaway spike; see 16-warehouse-tds.md) | 🟢 Real (reflection) |
| Warehouse — read-write T-SQL | Client CREATE/INSERT/SELECT relay straight to the sidecar; the warehouse owns its data (no reflection) | 🟢 Real (relay) |
Fabric SQL Database (database/) — OLTP + OneLake mirror | Same read-write TDS/FedAuth path (its own SQL Server database), plus mirroring: POST …/sqlDatabases/{id}/refreshMirror snapshots every table to OneLake as Delta (real Parquet + _delta_log), so Spark / DuckDB / delta-rs query the operational data. Verified with a go-mssqldb-writes → mirror → Delta-reads-back e2e (gated). Continuous/CDC mirroring and write-back-to-Delta are the deferred edge | 🟢 Real (snapshot mirror) |
| Per-item isolation (each item = its own SQL Server database) | Lakehouse/Warehouse routed by type; per-item databases so they never collide | 🟢 Real |
| RBAC → SQL permissions | Workspace role enforced on connect: no role → rejected; Viewer → read-only; Contributor+ → read-write (warehouse) | 🟢 Real |
information_schema / sys.* introspection | Relays natively — reflected/warehouse tables are real SQL Server tables | 🟢 Real (relay) |
| Per-column type fidelity (real SQL types over the wire) | The splice forwards SQL Server’s own COLMETADATA, so every column carries its true native type over the wire (the re-encode fallback, used only by fake test backends, synthesizes INTN/FLTN/BITN and falls back to NVARCHAR text) | 🟢 Real (native) |
Nested CTEs (WITH x AS (WITH y AS …)) | Fabric supports standard, sequential and nested CTEs; the SQL Server sidecar rejects the nested form (Msg 156), which would make the emulator stricter than Fabric. The TDS layer closes the gap on the wire: it parses the WITH prefix with a real lexer (string literals, quoted identifiers, nestable block comments) and flattens the nesting into the sequential form the sidecar accepts, then re-encodes the batch. Statements Fabric itself refuses are rejected rather than rewritten — non-SELECT targets, DML or OPTION hints in a nested definition, same-level duplicate names, and references escaping their nesting scope (the last of these verified to otherwise succeed post-flattening, i.e. a real divergence, not a theoretical one). Cross-level name shadowing is refused by name rather than renamed, because rewriting references safely needs a parser this layer deliberately is not. Statements sent as parameters (sp_prepexec/sp_executesql) are rewritten too: the RPC parameter list is walked, the statement parameter re-encoded, and the result re-parsed and compared with the original before it is forwarded — every other parameter byte-identical, or the original goes instead. Unmodelled parameter types (TEXT/XML/UDT/SQL_VARIANT) forward untouched rather than being guessed at. Witnessed by examples/medallion-pyspark running dbt’s native accepted_values and relationships — which compile to nested CTEs — with no CTE-free substitute in the project, and by a parameterized-statement probe through the real ODBC driver (29-tsql-parity.md) | 🟢 Real (rewrite + refusals) |
CTAS (CREATE TABLE … AS SELECT) | Fabric/Synapse spell “materialise this query” as CTAS; SQL Server has no such statement and spells it SELECT … INTO, so the sidecar rejects valid Fabric T-SQL with Msg 156. The TDS layer rewrites it, splicing INTO before the first FROM at paren depth 0 — including inside the EXEC('…') dynamic SQL dbt-fabric actually ships, which a statement-level rewrite would miss. Fabric’s WITH (DISTRIBUTION = …) options are dropped (physical layout, not results). Witnessed by examples/medallion-pyspark building gold with dbt’s own +materialized: table (29-tsql-parity.md) | 🟢 Real (rewrite) |
Class B strict mode (-tsql-strict) | Refuses what real Fabric rejects but SQL Server accepts — recursive CTEs, triggers, synonyms, CREATE USER, SET TRANSACTION ISOLATION LEVEL/ROWCOUNT/IDENTITY_INSERT, FOR XML, IDENTITY(seed, increment), enforced key constraints, multi-column statistics, PREDICT, sp_showspaceused — so a locally green build means a Fabric-green build. Off by default: it removes capability, which is the operator’s call. Indexed views and FOR JSON-in-a-subquery stay unenforced, with the reason recorded | 🟢 Real (opt-in) |
| Connection by item name (vs GUID) | Workspace read from the server name (<workspace>.datawarehouse.fabric.microsoft.com), item resolved by display name; a GUID still resolves by id (back-compat). Verified with a real go-mssqldb client | 🟢 Real |
| Fabric feature | Emulator | Type |
|---|
Data Pipeline control flow (If / ForEach / Until / Switch / Filter / Fail, expression language, dependsOn) | Pure-Go interpreter that really executes | 🟢 Real (orchestration) |
| Per-activity policy — retry + backoff + timeout | Applied to every activity type: policy.retry re-runs a failed activity (each retry from scratch; only the final outcome is recorded, carrying retryAttempt); policy.retryIntervalInSeconds is folded into the run’s durationInSeconds as virtual backoff; policy.timeout fails an attempt whose own virtual duration exceeds the limit. No real sleeping — backoff and timeouts are exercised in milliseconds on the controllable clock | 🟢 Real |
ForEach sequential / parallel (isSequential, batchCount) | Iterations run in array order (deterministic); the mode sets the reported wall-clock — sequential iterations add, a parallel batch costs its slowest — matching how real Fabric overlaps them | 🟢 Real |
List pagination (continuationToken) | On by default, as in Fabric: a list past the server page size returns an opaque continuationToken and an absolute continuationUri, with no client opt-in — so a client that ignores the token breaks here exactly as it would in production. ?maxPageSize may narrow a page, never widen it. The page size is a testing lever (-list-page-size / FABRIC_LIST_PAGE_SIZE): set it to 2 and every list forces the token loop on the spot, the same idea as the controllable clock for LROs. Witnessed by Microsoft’s fab CLI — a plain list (no page-size parameter) comes back paged, and paging through is asserted complete: no item twice, none missed, terminal page tokenless | 🟢 Real |
| Invoke pipeline (ExecutePipeline) | Resolves the referenced DataPipeline (GUID or name, optional other workspace) and runs it through a fresh interpreter — real recursive interpretation, one level deeper on the same engines. waitOnCompletion (default) gates the parent on the child’s terminal status; parameters flow into the child; a cycle or excessive nesting fails loudly | 🟢 Real |
| Pipeline → notebook activity (TridentNotebook) | Resolves the notebook reference and creates a real RunNotebook job instance the pipeline gates on — the pipeline→jobs linkage is real; the notebook’s cells execute on the Spark sidecar, and a job with cells outstanding has no clock-derived completion: it stays non-terminal until the engine reports, so a completed job means the cells ran. Cell execution is witnessed end-to-end by ci:medallion: the pipeline’s Notebook activity starts a genuine run, Sail executes it over Spark Connect, and the example asserts the engine-reported read/write set as the Notebook lineage edge — evidence that exists only if the cells ran, not inferable from the binding | 🟢 Real (chain + cells) |
queryactivityruns detail | Full | 🟢 Real |
| Activity-level lineage | Successful Copy execution persists its resolved workspace/item/path source→sink edge, returns it in activity output, and exposes workspace lineage for OpenMetadata ingestion | 🟢 Real (Copy) |
| Copy activity — OneLake → OneLake | Really moves the bytes through the storage layer: a file, or a directory subtree preserving structure; source/sink locations {workspaceId?, itemId, path} are expression-resolved (GUID or name); returns real filesWritten / dataWritten. External stores / format transformation are out of scope and fail loudly | 🟢 Real (in-family) / 🔴 external |
| Lookup activity — OneLake CSV/JSON/Parquet/Delta | Reads real rows from a CSV, JSON, or standalone Parquet file, or a lakehouse Delta table (Tables/<name>, auto-detected — no format hint needed) in OneLake; honors firstRowOnly; the result flows into @activity(…).output for downstream steps. Parquet/Delta reuse the warehouse’s own Parquet reader — a real Delta column keeps its native type (int/float/bool), not a stringified cell | 🟢 Real (CSV/JSON/Parquet/Delta) |
| GetMetadata activity — OneLake path | Stats a real OneLake path: exists / itemType / size / lastModified / childItems; a missing path honestly returns exists:false | 🟢 Real |
| Script / SqlServerStoredProcedure activities | Run real T-SQL against a Warehouse/Fabric-SQL-Database item’s own SQL Server database — the same per-item backend the TDS endpoint and the SQLDatabase mirror share. Script runs each scripts[] entry (Query → real rows back, NonQuery → rows affected); SqlServerStoredProcedure calls a real stored procedure with named parameters. The target is named directly as {workspaceId?, itemId} (the emulator’s own scoped mapping — real Fabric’s linkedService/connection reference isn’t modeled), the same shape Copy/Lookup/GetMetadata already use. Honest error without a warehouse SQL backend attached | 🟢 Real (scoped) |
| Web / WebHook activity | Makes the real HTTP call: method, URL, headers and body all resolved as expressions; a JSON response is merged into the activity output at the top level, so @activity('X').output.field resolves downstream; ADFWebActivityResponseHeaders and statusCode carried as Fabric does; a non-2xx fails the activity, matching Fabric rather than handing the status back; policy.timeout becomes the request deadline. Bounded at 8 MiB — the output lives in the run record, so it is not a download mechanism. FABRIC_WEB_ACTIVITY=stub restores the old record-without-calling behaviour for a hermetic CI leg, and labels the output stubbed: true so such a run cannot be mistaken for one that called | 🟢 Real |
REST connector — RestSource and RestSink in a Copy, with pagination | Makes the real HTTP requests and commits the rows: url (or the linked service’s url + dataset’s relativeUrl), requestMethod GET/POST, requestBody, additionalHeaders and httpRequestTimeout, all expression-resolved. The JSON response is shaped into a Delta table in the sink lakehouse — translator.collectionReference picks the record array, and a response holding exactly one array needs no hint; translator.mappings lift fields out of nested records by JSONPath — the shape BMC Helix actually returns (entries[].values.*), where auto-flatten finds no scalar columns at all. Witnessed end to end by e2e/rest-helix: a Web activity logs in for an AR-JWT, the copy pages limit/offset through a stand-in Helix that rejects Bearer, and the incidents land as Delta. paginationRules really page: AbsoluteUrl (absolute or relative) and QueryParameters.x / Headers.x cursors read out of the body (JSONPath) or the response headers; {placeholder} selectors stepped by RANGE:start:end:step (an empty end is open-ended) — the ServiceNow/BMC Helix limit+offset shape; EndCondition: with Empty / NonExist / Exist / Const:v; MaxRequestNumber; and RFC 5988 Link: …; rel="next", on by default only when no other rule is declared, as Fabric documents. The loop also stops on HTTP 204 and on any JSONPath resolving to null. A rule the emulator does not implement is refused by name rather than ignored. Accept is forced to application/json as Fabric does; a non-2xx fails the copy; a top-level JSON array with pagination rules is refused, since Fabric does not paginate that shape. Nested fields have no column shape and are reported by name in skippedColumns. Bounded at 8 MiB per page, 1M rows and 1000 pages — refused, never truncated, so a next that points at itself fails loudly instead of looping. Auth is Anonymous + additionalHeaders; the emulator models no connections, so any other authenticationType fails by name — a Web activity can fetch a token and pass it as an expression, which is how BMC Helix’s AR-JWT scheme works here. RestSink writes the other direction: rows from a Delta table or a Parquet/CSV file are POSTed (or PUT/PATCHed) in batches of writeBatchSize (default 10000), each request carrying Fabric’s payload — a JSON array of row objects. httpCompressionType: gzip, additionalHeaders, httpRequestTimeout and requestInterval (validated against Fabric’s [10, 60000]ms band, and virtual like retry backoff) are honoured; a non-2xx fails the copy, zero rows send no request rather than an empty array, and a BinarySource is refused because opaque bytes have no rows to send | 🟢 Real |
Salesforce — SalesforceV2Source and SalesforceV2Sink in a Copy | Runs the real Bulk API 2.0 query lifecycle: creates a query job (operation is queryAll when includeDeletedObjects is set — a different operation, not a filter), polls it to a terminal state, then downloads CSV result sets paged by the Sforce-Locator header, which ends on the literal string "null". objectApiName or a SOQL query; with neither, all the object’s data. A Failed/Aborted job fails the copy naming the job id so it is findable in the org’s own monitor. reportId is refused by name — a report is the Analytics REST API, not a Bulk query, and running a query instead would return the object’s rows rather than the report’s. The org is named directly (instanceUrl + accessToken, both expression-resolved) because the emulator models no connections; a Web activity can run the OAuth call and pass the token in. Bounded at 8 MiB per page, 1M rows and 1000 pages. Sink (SalesforceV2Sink) is scoped, not built — 41-salesforce-connector-plan.md. SalesforceV2Sink writes the other direction through the ingest lifecycle: create a job, PUT the CSV to its contentUrl, PATCH it to UploadComplete (a job left Open uploads rows nothing ever processes), then poll. One job per writeBatchSize (default 100,000); writeBehavior Insert or Upsert, with externalIdFieldName required for upsert and refused without it. ignoreNullValues picks between Bulk CSV’s two meanings — an empty field leaves the value unchanged, the literal #N/A sets it to NULL — and an empty string stays an empty string rather than becoming either. A job that reaches JobComplete with records rejected fails the copy, naming failedResults, because a partial write reported as a whole one is the worst outcome available. Witnessed end to end by e2e/salesforce: a round trip — OAuth via a Web activity, five accounts out over three locator pages into Delta, then upserted back in three ingest jobs, with every record compared against what the org originally served | 🟢 Real |
| External-connector leaves (ServiceNow, …) | Stubbed success — reached in dependsOn order with inputs resolved, but nothing executes: each needs a vendor SDK and credentials the emulator has neither of. The REST connector above is the working route to any of them that expose a REST API — and to BMC Helix, which real Fabric has no connector for at all | 🟡 Emulated |
| Apache Airflow Job | Typed item + beta file APIs; uploaded Python DAGs sync to an opt-in real Airflow 2.10.5/Python 3.12 sidecar, whose scheduler/executor and REST state determine the Fabric job result (e2e/airflow) | 🟢 Real (sidecar) |
| Dataflow Gen2 (Power Query M engine) | Typed item/definition management round-trips. Refresh, Publish, and in-pipeline execution fail with DataflowEngineNotImplemented; no open Power Query M engine exists to attach | 🟡 mgmt / 🔴 exec |
| Connectors / on-prem gateways | — | 🔴 Not implemented |
| Fabric feature | Emulator | Type |
|---|
| Git integration (connect / status / commit / update / disconnect) | Full, real state | 🟢 Real |
fabric-cicd tool publishing | The real client round-trips definitions (e2e) | 🟢 Real |
| Deployment pipelines — model, assignment, item pairing, Deploy Stage Content (D0–D2) | Real promotion: definitions really copy, pairs decide (not names), metadata only — a deployed lakehouse arrives empty — and target-only items survive. 202 LRO + /result detail. 23-deployment-pipelines.md | 🟢 Real |
| Deployment pipelines — role-assignment CRUD (D3) | Add / Delete / List; Admin is the only role a pipeline defines; mutations require Admin, reads require membership | 🟢 Real |
| Fabric area | Emulator | Type |
|---|
Real-Time Intelligence — Eventhouse / KQL Database (real-time-intelligence/) | Full item management (including the default child database an eventhouse creates, and creationPayload.parentEventhouseItemId), plus the Kusto REST protocol on the eventhouse’s published properties.queryServiceUri — /v1/rest/mgmt, /v1/rest/query, /v2/rest/query — terminated by the emulator (Kusto-audience bearer, workspace RBAC, one isolated engine database per Fabric KQL Database) and executed by Microsoft’s own KQL engine container (kustainer) when the rti profile attaches it. No engine attached → honest 501. 25-rti-kusto.md | 🟡 mgmt / 🟠 exec (opt-in --profile rti) |
Real-Time Intelligence — Eventstream (real-time-intelligence/event-streams/) | Item management only. The attached Kusto engine is a query/ingest engine with no streaming ingestion — a streaming pipeline is a different service, deferred with cause | 🟡 mgmt / 🔴 exec |
CopyJob, KQLDashboard, KQLQueryset, WarehouseSnapshot | Typed collections over the generic item surface — create/get/list/patch/delete and definition round-trip, with the collection forcing its type. Type names taken from fabric-docs payloads; no execution engine for any of them (a Copy Job does not copy) | 🟡 mgmt / 🔴 exec |
Event triggers — Reflex (Data Activator) on OneLake events | The Reflex triggers. A trigger subscribes to a source item’s OneLake file events (FileCreated / FileDeleted / FileRenamed) under an optional path prefix, and a match starts a real item job — invokeType: "EventTriggered" — with the event bound in as @pipeline()?.TriggerEvent?.FileName / .FolderPath / .Subject, reading which needed safe navigation (?.) in the expression language, so one definition works whether or not a trigger started it. No broker is needed: every byte written to OneLake passes through the emulator’s own storage layer, so a file event is observable at the source whoever wrote it — an ADLS client, azcopy, delta-rs, a Copy activity, the mirror writer. Dispatch is synchronous and reentrant, so bronze→silver→gold chains work; a runaway is bounded by an activation count rather than by identity, matching how Fabric bounds one — Activator documents Fabric item — Activations/user/minute — 50 and throttles or cancels beyond it, and caps input at 10,000 events/second/rule, with no documented loop detection or dedup. Independent events each activate, because Activator “continues monitoring without waiting for the action to complete”. Boundary: the binding is an emulator-native control surface (…/reflexes/{id}/triggers) because Fabric has no public REST for it — the Eventstream/Reflex rule is assembled in the portal. What is faithful is everything downstream: the filter, the invocation, the TriggerEvent fields, and a real pipeline really running. 07-control-plane-api.md | 🟢 Real (OneLake sources) |
Mirroring — Mirrored Database (mirroring/) | POST …/mirroredDatabases/{id}/refreshMirror mirrors an external SQL Server source (reached via a Connection with Basic credentials) to OneLake as real Delta — reusing the exact same mirror writer the Fabric SQL Database uses (warehouse.Mirror; same code, external source). Proven by a gated e2e: a table seeded directly on an external database (bypassing the emulator’s own per-item routing entirely) mirrors and reads back correctly. Snapshot-on-trigger, not continuous/CDC replication; other source engines (Snowflake, CosmosDB, on-prem via gateway) are out of scope | 🟢 Real (snapshot mirror, SQL Server sources) |
Power BI — Semantic Model query (executeQueries) | Real bounded DAX engine — EVALUATE, SUMMARIZECOLUMNS, measures, SUM/DIVIDE/COUNTROWS/IF/SELECTEDVALUE, the infix operators (+ - * / &, comparisons) with DAX precedence, relationship filter propagation, and a name that does not resolve erroring rather than aggregating to zero — over imported data.json or compatibility-level-1604 Direct Lake entity partitions backed by current OneLake Delta. Proven by the golden DAX/GX suites and the Spark-written Direct Lake witness. | 🟢 Real (DAX subset + Direct Lake) |
| Power BI — Reports / rendering; full DAX; SemPy over XMLA | No report rendering; DAX beyond the fixture subset; and the native ADOMD.NET/XMLA transport SemPy uses (no CI oracle) — all deferred with cause | 🟡 mgmt / 🔴 render |
Data Science — ML models / experiments / MLflow (data-science/) | Authenticated, workspace-scoped proxy to a real MLflow 3 tracking/model-registry server. Experiment/model creation synchronizes typed Fabric items; experiment/run references are isolated by workspace; successful artifact uploads are mirrored under the experiment item’s OneLake Files/mlflow-artifacts. | 🟢 Real (sidecar) |
Graph (graph/), Real-Time Hub, Copilot / IQ (iq/), Embed, Workload Dev Kit | — | 🔴 Not implemented |
| Capability | Purpose |
|---|
Controllable clock (/_emulator/clock) | Advance virtual time to drive LRO / job status transitions deterministically. |
Fault injection (/_emulator/faults, /_emulator/permissions) | Force failures / throttling / RBAC denials to test client resilience. |
Flow stream (/_emulator/events) | Server-Sent Events for every byte that moves through OneLake — file events for writes/renames/deletes whoever made them, and table events derived from Delta commits (version, rowsAdded, filesAdded). Plus job and activity events — a failing activity reaches the stream with its error and job id the moment it fails, rather than being reconstructed from queryactivityruns afterwards — and attribution: a Copy activity’s writes name the activity, a notebook cell’s name the cell (including engines that cannot set headers, via the same bearer claims lineage uses). Never inferred; a write with nothing to say carries no attribution. curl -N tails a medallion run live; ?kinds= narrows it and ?since= replays the ring buffer. Delivery is deliberately lossy — a slow consumer is told what it dropped rather than being allowed to stall a writer. Two further kinds complete a medallion’s chain: lineage when a movement is recorded (so a graph redraws as a warehouse build happens, which has no job to end), and query when a semantic model is read — the Power BI hop, which is an event and never an edge because a query moves no data. 31-flow-observability.md |
| Data-flow lineage beyond OneLake | A medallion’s last hops move no bytes through OneLake, so each is observed on its own terms rather than inferred. silver → gold: the TDS front already parses every statement for dialect adaptation, so it also records what the engine accepted — CTAS, SELECT … INTO, INSERT … SELECT, CREATE VIEW, sp_rename, DROP, including inside the EXEC('…') dbt-fabric actually ships — resolving three-part names to Fabric items (producer: Warehouse). dbt’s build-then-swap is followed through sp_rename, so the graph names fct_order_lines and not its __dbt_temp scaffold. A statement whose response carries an error, or whose outcome cannot be read, records nothing. gold → semantic model: a Direct Lake table’s binding names its source, so the edge is recorded when the definition lands (producer: DirectLake); an import model gets none, because its rows arrive detached from wherever they were selected and inventing a source would be a guess. Interactive engines: POST /v1/workspaces/{wid}/lineage is the notebook read/write report without the job, for a Spark session or plain script (producer: Reported) — emulator-native, as Fabric has no such endpoint. The producer is the point: Warehouse is evidence, Reported is a claim, and a consumer never has to guess which. 31-flow-observability.md |
| Svelte management portal | Dashboard, workspaces, operations, clock, and fault controls. |
Parity isn’t claimed from our own tests alone — each 🟢 surface is pinned against
the real, unmodified client a Fabric user runs, executed against the emulator
in CI (e2e/<client>/). If Microsoft’s own tool round-trips unchanged, the
contract holds better than any assertion we could write ourselves.
| Real client (pinned) | Surface exercised | Status |
|---|
fabric-cicd (Microsoft) | Control plane / CI-CD publish | 🟢 e2e/fabric-cicd |
Fabric CLI fab (Microsoft) | Control plane — SPN auth (MSAL) + workspace/item CRUD (Notebook, SemanticModel, Report, DataPipeline, Lakehouse), ls/get/api | 🟢 e2e/fabric-cli |
| Fabric Data Engineering VS Code extension 1.18.1 contract (Microsoft) | Shared-backend/MWC authoring routes through api.powerbi.com; interactive kernel websocket is not claimed | 🟢 e2e/vscode-extension |
| Apache Airflow 2.10.5 | ApacheAirflowJob DAG discovery, scheduling, execution, and status | 🟢 e2e/airflow |
| MLflow 3 + dbt-duckdb | Workspace-scoped experiment/run/artifact/model lifecycle, followed by dbt’s real Delta plugin over the same Spark-written OneLake table | 🟢 e2e/data-science-loop |
deltalake (delta-rs) | OneLake Delta write/read | 🟢 e2e/delta-rs |
azure-storage-file-datalake + Blob SDK | OneLake ADLS Gen2 DFS + Blob | 🟢 e2e/adls-sdk |
azcopy (Microsoft) | OneLake Blob multi-block transfer | 🟢 e2e/azcopy |
| DuckDB | Lakehouse SQL over Delta/Parquet | 🟢 e2e/duckdb |
| PySpark behind the Livy API | Spark sessions / statements | 🟢 e2e/spark, e2e/livy, e2e/notebook-run |
notebookutils | Notebook utility shim | 🟢 e2e/notebookutils |
go-mssqldb | Warehouse/Lakehouse TDS + FedAuth | 🟢 internal/server, internal/tds |
dbt-fabricspark (Microsoft) | Fabric Spark via Livy HC sessions | 🟢 e2e/dbt-fabricspark — debug→seed→run→test on Sail |
dbt-fabric (Microsoft) | Warehouse TDS via ODBC Driver 18 | 🟢 e2e/dbt-fabric — debug→seed→run→test through the TDS splice |
azure-kusto-data (Microsoft) + raw Kusto REST, over kustainer (Microsoft’s own KQL engine) | Eventhouse / KQL Database: /v1/rest/mgmt, /v1/rest/query, /v2/rest/query on the published queryServiceUri — create table, ingest, query values back, per-database isolation | 🟠 e2e/rti — witness of record is CI (amd64). The engine needs AVX2, which Rosetta does not provide, so the default Docker setup on Apple silicon cannot run it; a QEMU x86-64 VM with --cpu-type max can, and does (25-rti-kusto.md) |
The TDS surface now has two independent driver witnesses: go-mssqldb and
the Microsoft ODBC Driver 18 (via dbt-fabric). That second driver mattered —
it exposed a real gap: go-mssqldb tolerated a synthesized FedAuth login, but
ODBC Driver 18 took a compatibility path (prepared-statement RPCs +
sp_reset_connection under mandatory connection pooling) that desynced against a
re-encoding relay. The fix was to byte-splice the post-login session straight
to the real SQL Server (so it emits every token itself), which is exactly the
kind of driver-family gap a single-driver test never surfaces. dbt-fabricspark
likewise drives the high-concurrency Livy layer over its real Livy-session
protocol (method: livy, service-principal auth via entra-emulator).
The emulator targets Microsoft Fabric — the convergence/successor product —
not the earlier Azure analytics services Fabric replaced. That boundary is why
some adjacent dbt adapters and Azure surfaces are intentionally not built:
they belong to predecessor (often retired) products, and their Fabric-native
successors are what we emulate instead.
| Adjacent product / client | Why out of scope | Fabric-era equivalent (in scope) |
|---|
Azure Synapse dedicated SQL pool (dbt-synapse) | Different product: its own control plane (Synapse workspaces) and an MPP T-SQL dialect (DISTRIBUTION = HASH, clustered-columnstore / resource-class DDL) that our vanilla SQL Server sidecar rejects. dbt-synapse layers on dbt-fabric, so the shared SQL path is already covered by the dbt-fabric witness | Fabric Warehouse — 🟢 TDS relay |
Azure Data Lake Analytics — U-SQL / SCOPE (dbt-scope) | Retired service (EOL Feb 2024), proprietary batch language, no Fabric embodiment. The only overlap (Delta on a lake) is Spark/OneLake, already witnessed | Fabric Spark — 🟠 Livy |
| ADLS Gen1 | Retired (Feb 2024), superseded by Gen2 | — |
| ADLS Gen2 (standalone storage account) | Not missing — OneLake is the Gen2 endpoint: hierarchical namespace, the dfs filesystem API, onelake.dfs.fabric.microsoft.com. Fabric has no separate storage account to emulate | OneLake — 🟢 e2e/adls-sdk |
Rule of thumb: if a capability exists only in a product Fabric replaced, it’s out
of scope; its Fabric-native successor is what we build. “We already have the
TDS/SQL Server foundation” makes Synapse cheaper, not done — the remaining
delta is a whole MPP dialect plus a second control plane, for a superseded
target. So the two dbt adapters we build (dbt-fabricspark, dbt-fabric) are
exactly the two that hit live Fabric surfaces; the other two (dbt-synapse,
dbt-scope) target predecessor products outside the emulator’s remit.
Real Fabric’s own Livy endpoint is Microsoft’s implementation of the Livy REST
contract over their Spark platform — they honor the protocol, not the retired
Apache Livy server. And where Fabric adds its own layer on top of that
protocol — high-concurrency REPL packing, which a vanilla Livy server has no
concept of — the emulator implements that layer directly rather than proxying,
because there is nothing to proxy it to. That is the same stance throughout: the
protocol and control plane are the durable, real things (built, not mocked,
so real clients can’t tell the difference), and the compute engine is attached
(Spark; T-SQL on SQL Server; KQL on Microsoft’s own Kusto engine —
25-rti-kusto.md) or deferred when proprietary or without an
implementation to attach at all (Dataflow Gen2’s M engine, Power BI rendering,
Eventstream’s streaming ingestion). Every deferral fails loudly rather than
pretending to succeed. See 13-roadmap.md for the milestone
history and the deferred-with-cause rationale.