How the emulator’s surface maps to real Fabric (as documented at
learn.microsoft.com/fabric /
MicrosoftDocs/fabric-docs), and
— the point of this table — whether real work happens or just the API shape.
The emulator’s design bet is that the durable, testable surface is
contracts + storage + identity + orchestration, and those are done for real
(real signed JWTs, real Delta bytes on disk, real RBAC, a real pipeline
interpreter, real cross-engine SQL, real Livy high-concurrency session packing).
The heavyweight or proprietary compute engines run by default: docker compose up auto-loads an override that starts Sail (Spark Connect) and a SQL
Server sidecar, so Livy sessions, notebook cells and the T-SQL warehouse do real
work out of the box. What stays 🟠 is the narrower set that needs a different
engine — the JVM overlay, or an opt-in profile — and what cannot be done
honestly at all is stubbed.
“Real via our own wire-protocol implementation.” A row is 🟢 Real not
only when an external engine/client does the work, but also when the emulator
itself implements Fabric’s wire protocol and the logic behind it — so a real,
unmodified client gets byte- and behaviour-identical responses. Fabric’s
control plane, OneLake’s ADLS/Blob surfaces, the Data Pipeline expression
language + control flow, and the Livy high-concurrency session-packing layer
are all in this category: no engine is being proxied, yet the observable
contract matches real Fabric because we built the protocol, not a mock of it.
Where a row’s execution needs an engine that is not the default — JVM-only
Spark surfaces, an opt-in KQL profile — that part is split out as 🟠 or 🔴.
| Meaning |
|---|
| 🟢 Real | Genuine work: real signed JWTs, real bytes on disk, a real engine/client computes, real logic enforced — no pretending. |
| 🟡 Emulated | Faithful API contract + persisted state, but no engine — status is clock-derived / management-only. |
| 🟠 Non-default engine | Real, but only on an engine that is not the default: the JVM Spark overlay (docker-compose.spark-jvm.yml) or an opt-in profile such as --profile rti. Not “bring your own”: docker compose up already starts Sail and the SQL Server sidecar, so the everyday Spark and T-SQL surfaces are 🟢. |
| 🔴 Not implemented | Honest 501 or absent. |
| Fabric feature | Emulator | Type |
|---|
| Workspaces CRUD | Full. Display names are unique tenant-wide — duplicates 409 WorkspaceNameAlreadyExists (uniqueness per the REST reference; fabric-docs covers workspace naming portal-side only) | 🟢 Real |
| Items CRUD + typed collections | Full. Display names are unique per (workspace, type) — duplicates 409 ItemDisplayNameAlreadyInUse; names stay reusable across types, which is why OneLake addresses items as name.Type. The type is validated against the documented ItemType enumeration (50 values, REST reference) — anything else is InvalidItemType, as real Fabric returns — and canonicalised case-insensitively so notebook and Notebook cannot become two types. 24 typed collections alias the generic surface; each collection segment is taken from its own reference page, since they are not derivable (GraphQLApis is capitalised, variableLibraries is not) | 🟢 Real |
| Role assignments / workspace RBAC | Enforced from the validated bearer principal | 🟢 Real |
| Folders | Full | 🟢 Real |
| Capacities (list, assign / unassign) | Full state, no billing/SKU enforcement | 🟢 Real state |
| Long-running operations (202 → poll) | Clock-derived | 🟡 Emulated |
Item job execution (jobs/instances) | Generic items: status clock-derived. DataPipeline jobs really run the interpreter (see Data Factory) and set terminal status from the run | 🟡 Emulated / 🟢 Real (pipelines) |
List Item Job Instances (GET …/jobs/instances) | An item’s runs, paged, newest first, with the same clock-derived status the single-instance read returns. Listing is itself an evaluation point for the scheduler below, so a run that has come due appears without touching the control surface | 🟢 Real |
Item Job Scheduler (…/items/{id}/jobs/{jobType}/schedules) | Fabric’s own per-item scheduler — distinct from the ApacheAirflowJob item, which delegates to a real Airflow sidecar. All four ScheduleConfig members: Cron (interval 1–5,270,400 min), Daily/Weekly (times[] ≤ 100, weekdays[]), Monthly (DayOfMonth or ordinal weekday, recurrence 1–12) — every documented bound enforced at write time, the 20-per-item ceiling as ScheduleExceedsLimit, and localTimeZoneId (Windows ids or IANA) honoured as real local wall time, so a daily 09:00 stays 09:00 across a DST change. Schedules really fire: each due occurrence starts a job instance down the same path a manual run takes, so a scheduled DataPipeline executes the interpreter and only invokeType: "Scheduled" tells them apart. Evaluation is driven by the controllable clock rather than a background worker — advancing time materialises exactly the occurrences that came due, and a past startDateTime triggers instantly with no special case. Boundary: catch-up is capped at 100 occurrences per evaluation (newest kept), because a controllable clock can be advanced a year against a one-minute Cron. 07-control-plane-api.md | 🟢 Real |
| Fabric feature | Emulator | Type |
|---|
| Entra OAuth2 tokens / JWKS / client-credentials | entra-emulator mints real signed JWTs | 🟢 Real |
| Workspace managed identity handshake | Provisioned via entra admin API; the identity’s own token passes RBAC | 🟢 Real |
| Key Vault references in connections | Resolved against azure-keyvault-emulator | 🟢 Real |
Governance domains (/v1/admin/domains) | Full admin surface: domain/subdomain CRUD (the hierarchy stops at two levels, as documented), workspace assignment (single-valued — re-assigning moves a workspace), nonEmptyOnly listing, and bulk Admins/Contributors role assignment. Deleting a domain cascades to its subdomains, assignments and roles. No tenant-admin gate: the emulator has no Fabric-administrator role model, so any authenticated principal may call these | 🟢 Real (mgmt); 🟡 tenant-admin gate |
Activity log / audit (GET /v1.0/myorg/admin/activityevents) | Real audit trail — the emulator records events as operations happen, using the documented audit vocabulary (CreateWorkspace, CreateArtifact/UpdateArtifact/DeleteArtifact from admin/operation-list.md; InsertDataDomainAsAdmin & co. with their DataDomainObjectId/FoldersToSetCounter properties from governance/domains-audit-schema.md). Enforces the documented request rules (single-quoted UTC bounds, same-day window) and pages with continuationToken/continuationUri until the token stops coming back. Nothing is synthesised at read time | 🟢 Real |
Tenant settings (GET /v1/admin/tenantsettings) | The documented TenantSetting object in full — settingName, title, enabled, canSpecifySecurityGroups, tenantSettingGroup, the three delegateTo* flags, enabled/excluded security groups (graphId/name), and typed properties validated against the documented TenantSettingPropertyType enum. Optional arrays are omitted, not null, as the reference’s sample does; seeded with the setting names that sample uses. POST /v1/admin/tenantsettings/{name}/update is the real update API (enabled required; response wrapped as {"tenantSettings":[…]}) | 🟢 Real |
Tenant-wide workspace admin (GET /v1/admin/workspaces) | The documented Workspace shape — note it differs from the user-facing surface: the envelope key is workspaces (not value) and the field is name (not displayName). Filters type/state/capacityId/name are enforced, with undocumented enum values returning BadRequest as the reference specifies; domainId is reported from the real domain assignment. Emulator workspaces are always Workspace/Active (no soft delete), so state=Deleted is legitimately empty. No tenant-admin gate — as with the other admin routes | 🟢 Real (mgmt); 🟡 tenant-admin gate |
Tenant-wide item admin (GET /v1/admin/items) | The documented Item shape across every workspace, with workspaceId/capacityId/type/state filters and the reference’s own error codes (InvalidItemType, InvalidItemState). Envelope key is itemEntities — a third spelling after the user-facing value and admin workspaces’ workspaces, which is why each came from its own reference page. Active is the only documented state, so that is all the emulator reports. Fields it does not model (creatorPrincipal, defaultIdentity, tags) are omitted rather than faked | 🟢 Real (mgmt); 🟡 tenant-admin gate |
Capacity tenant-setting overrides (GET /v1/admin/capacities/delegatedTenantSettingOverrides, POST …/{capacityId}/delegatedTenantSettingOverrides/{name}/update) | The documented CapacityTenantSetting (a tenant setting plus delegatedFrom and delegateToWorkspace, and without delegateToCapacity/delegateToDomain). The update body is the documented one — enabled required, no properties, no delegatedFrom — and the response wraps as {"overrides":[…]}. An override may only be created for a setting whose delegateToCapacity is true. Note there is no /v1/admin/capacities list API in Fabric: capacities are listed on the Core surface at /v1/capacities | 🟢 Real (mgmt); 🟡 tenant-admin gate |
Sensitivity labels (bulkSetLabels / bulkRemoveLabels) | The documented admin bulk APIs, reporting per-item successfulItems/failedItems rather than failing whole calls. Every change writes the documented SensitivityLabelEventData to the audit log — SensitivityLabelApplied/Changed/Removed with SensitivityLabelId, OldSensitivityLabelId, ActionSource 3 (Manual), ActionSourceDetail 5 (PublicAPI), ArtifactType 12, and a LabelEventType genuinely computed from label order (upgraded/downgraded/same-order/removed). The label taxonomy is emulator-provided — real Fabric gets labels and their order from Purview, which cannot be attached offline; GET /v1/admin/labels exposes it | 🟢 Real (label APIs + audit) / 🟡 taxonomy |
| Sensitivity labels → catalog | Labels are Purview Information Protection objects, not Atlas entities, so the Atlas-API route a Purview → OpenMetadata migration takes cannot carry them — assets, classifications, glossary and lineage cross; labels do not. The optional OpenMetadata profile exports them instead: the taxonomy becomes classification FabricSensitivity (mutuallyExclusive, since an item carries at most one label), each label a tag under it, and an item’s sensitivityLabel is applied to that item’s catalog entity. CI reads the tag back through OpenMetadata’s API, on two items with different labels, and asserts that clearing the label in Fabric clears the tag in OM (22-openmetadata.md) | 🟢 Real (stored in OM) |
Purview scanning / classification (governance/) | Purview itself is not attachable offline. What the emulator does model — labels, domains, the audit trail — reaches a catalog through the OpenMetadata profile (row above); scanning, classification rules and the Data Map API are not implemented | 🔴 Not implemented |
| Lineage (catalog graph) | Via the optional OpenMetadata profile: OneLake shortcut edges and executed pipeline Copy source→sink edges are persisted exactly and witnessed in OM’s graph API. Notebook edges come from the engine’s own report of what it read and wrote, or from the data plane observing it; Script/stored-procedure code is still not guessed. Catalog SSO can also be pointed at entra-emulator (22-openmetadata.md) | 🟢 Real (shortcuts + Copy) |
| Fabric feature | Emulator | Type |
|---|
| ADLS Gen2 DFS surface (create → append → flush, ranged read, list) | Full, incl. the x-ms-range dialect | 🟢 Real (real bytes) |
| Blob surface | Full | 🟢 Real |
| Delta commits (put-if-absent atomicity) | Real; -race-tested concurrent-commit race | 🟢 Real |
| Shortcuts (OneLake → OneLake) | Symlinks with target-side RBAC (trusted-workspace-access) | 🟢 Real |
| Shortcuts to external targets (S3 / ADLS Gen2 / Dataverse) | Real Amazon S3 / S3-compatible read-through: a Connection carrying the Access Key Id + Secret Access Key pair — as Basic credentials, because Fabric’s S3 connector uses authentication kind “Access Key” while the REST reference’s CredentialType enum has no AccessKey member and Basic is its only two-secret type — makes the emulator sign upstream requests with AWS SigV4 (internal/awssig, verified against AWS’s own published example signature). Witnessed by e2e/s3 against a real SeaweedFS server started with an identity config — the suite proves an unsigned GET is 403 and that a wrong secret is refused, so the pass cannot be vacuous; the object itself is written by boto3. ADLS Gen2 reads are witnessed too: e2e/azurite-shortcut drives a real SAS through the shortcut against Azurite, Microsoft’s own storage emulator, proving an unauthenticated GET is 403 and a tampered SAS is refused. Scope stated honestly — Azurite implements Blob/Queue/Table and not ADLS Gen2, so the endpoint is the Blob one; that witnesses the read path (a plain authenticated GET, identical on both endpoints of a real ADLS Gen2 account) but not DFS-specific behaviour, which stays unwitnessed offline. Dataverse remains an explicit 501. S3 read-only is by design — Fabric documents S3 shortcuts as read-only regardless of permissions, so a write there is refused locally and never reaches the target. ADLS Gen2 write-through is real: a file flushed under an ADLS shortcut is PUT to the storage account and the local buffer dropped (it would otherwise shadow later target changes), and deleting a file within a shortcut deletes it at the target, as documented. Witnessed by e2e/azurite-shortcut, which reads the result back with the Azure SDK straight from Azurite rather than trusting the emulator’s own 200 | 🟢 Real (S3 SigV4 + ADLS reads) / 🔴 Dataverse |
Engine-level claims below are measured, not asserted: the generated
Spark engine matrix probes each capability against both
Sail and the JVM overlay and is regenerated in CI.
| Fabric feature | Emulator | Type |
|---|
| Lakehouse item + Tables/Files storage | Full (via OneLake) | 🟢 Real |
| Notebook authoring / definition round-trip | Full | 🟢 Real |
notebookutils / mssparkutils (fs, credentials, getSecret, lakehouse, runtime) | Functional stdlib shim (python/notebookutils) | 🟢 Real |
| Spark session / statement / batch via the Livy API | Native termination (--spark-agent-url): the emulator implements the Livy contract and drives a persistent statement-executor agent. The default agent is a PySpark Connect client of Sail, not Apache Spark. An external Livy backend remains configurable with --spark-livy-url | 🟢 Real (default engine) |
| Notebook cell execution | The emulator parses and records the notebook run, resolves attached lakehouse/Environment metadata, and Sail executes cells by default. The same fixture runs on Spark 3.5 JVM and proves unqualified saveAsTable/spark.table bind to OneLake Tables/ | 🟢 Real (default engine; per-capability detail in engine-matrix.md) |
| Livy High-Concurrency (5-REPL) sessions | Fabric’s packing layer is implemented directly: sessionTag packing, 5-REPL cap + spill, independent lifecycle and slot reuse. Statements use Sail by default, so engine compatibility is limited to the Spark Connect subset | 🟢 Real (default engine) |
| Environments | Run binding resolves Python requirements, Spark properties, and JAR declarations. Python packages are provisioned per run; config is applied to the real session; JAR-bearing runs explicitly require JVM Spark | 🟢 portable subset / 🟠 JAR-bearing runs need the JVM overlay |
| Spark Job Definitions | V1 definition/main/arguments/libraries parsing, attached lakehouse+Environment resolution, Pending→Completed/Failed callback lifecycle, and real Sail/JVM execution witness | 🟢 orchestration / 🟡 engine execution witnessed at the parse layer only |
The engine behind the agent is Sail (Rust
Spark-Connect, no JVM). Every row is probed in CI (e2e/sail), not
inferred — the fidelity deltas a Fabric notebook author actually hits.
A Spark 3.5 JVM image (docker/spark-runtime, Fabric Runtime 1.3’s engine
baseline) exists as a CI compatibility oracle (e2e/spark-jvm), and the
statement agent still has its classic-session path. It is also exposed as a
user-facing overlay — docker compose -f docker-compose.yml -f docker-compose.override.yml -f docker-compose.spark-jvm.yml up swaps the
statement agent onto it — so the JVM-only rows below are graded 🟠
(🟠 non-default engine): real with that overlay attached, unavailable on the default
engine. Verified live, not inferred: sc.parallelize([1,2,3,4]).map(x*2) .sum() returns 20 through the Livy agent, and spark._jvm.io.delta.tables .DeltaTable resolves.
| Notebook pattern | Emulator (Sail) | Type |
|---|
abfss://…@onelake.dfs.fabric.microsoft.com/… production paths | Work unmodified (endpoint override routes the Hadoop URL form) | 🟢 Real |
| Delta write/read/append; SQL over temp views | Full | 🟢 Real |
Time travel — DataFrame option and SQL VERSION AS OF | Both forms work on Sail. The SQL form was previously graded 🔴 on the strength of an assertion; the generated engine matrix probes it against both engines and it passes on each | 🟢 Real |
MERGE INTO | Works against a registered table target (CREATE TABLE … USING delta LOCATION); path-based delta.`az://…` merge targets don’t resolve | 🟢 Real (registered) / 🔴 path target |
createDataFrame(local_rows) | Works (runners preset localRelationSizeLimit) | 🟢 Real |
sc / RDD API / spark._jvm | Fidelity inversion: works on real Fabric, impossible on Spark Connect — the agent binds sc to a guide-rail stub that raises a clear pointer instead of NameError. Restored by the JVM overlay (classic session): sc.parallelize(…).map(…).sum() verified | 🔴 default / 🟠 JVM overlay |
DML row-count envelopes (INSERT/MERGE counts) | Statement executes; DataFusion’s uint64 count is absorbed as an empty result by the SQL agent | 🟡 Emulated envelope |
| Structured streaming | Partially supported on Sail v0.6.6, and the detail matters: readStream plans (its schema resolves) and the streaming query manager works, but neither is evidence that data flows: a readStream is unobservable without a sink, and the console sink writes to the server’s stdout where no client can see it — driving it for six seconds produced no output at all. Sail also reports no progress metrics (lastProgress is None), so there is nothing else to interrogate. No durable sink works, and those are the ones whose output can actually be verified. memory → No table format found for: memory; parquet/csv → cannot write streaming data to listing table; delta → unsupported extension node for streaming: DeltaWriteNode. Since a streaming job that cannot land data is not useful, this stays 🔴 for real work. The engine matrix’s streaming rows now assert that rows reach the sink — files read back, or a queryable table — rather than that a query object reported itself active, which is what previously made console look green. Full support on the JVM overlay | 🔴 durable sink / 🟠 JVM overlay |
OPTIMIZE / VACUUM (Delta maintenance) | Partly real on the default engine, via delta-rs: Sail’s planner has no such commands, so the Livy agent recognises those statements and runs them through delta-rs against the same table — verified compacting 3 files into 1 and vacuuming the superseded 3, rows intact. ZORDER/WHERE are refused, not ignored (they change what is compacted); use the JVM overlay for the full syntax. Proven against a real abfss://…onelake… table end-to-end through Livy REST (e2e/livy), with a negative control: the same statement with credentials forced off is refused, so the pass is not vacuous. delta-rs authenticates on its own account — a Connect client cannot read back the session’s bearer — minting from the same issuer with the same Storage audience, resolved per statement so a refreshed token needs no restart. One caveat remains, unchanged and by design: the executor differs from real Fabric (the emulator performs it, not Spark). See 20-lakesail-engine.md | 🟢 Real (delta-rs) |
Java/Scala UDFs, spark.jars | Out of scope for Sail by design, not a technical impossibility. Sail already embeds a foreign runtime — it links CPython via pyo3 (sail-python-udf) for Python UDFs — so a JVM could be embedded too. The real obstacle is that a Spark Java/Scala UDF is compiled against Spark’s own classes (InternalRow, Catalyst encoders), so running one needs Spark’s jars loaded — which is JVM Spark, exactly what Sail exists to avoid. Bytecode translation (GraalVM, TeaVM) does not help: the problem is Spark’s API surface, not executing bytecode. The JVM overlay is the answer | 🔴 default / 🟠 JVM overlay |
| Change Data Feed | Read is real on the default engine: spark.delta_change_feed(uri) serves a genuine change feed from OneLake through delta-rs (verified in e2e/livy, 2 versions → 2 rows). Writing a CDF-enabled table is not possible on Sail — its writer answers Unsupported table features required: [ChangeDataFeed] even with the property and the feature both named, so the table has to come from delta-rs or the JVM overlay. Exposed as a named helper rather than an interception of spark.read, so it is always obvious which engine answered | 🟢 read / 🔴 Sail-authored table / 🟠 JVM overlay |
spark.jars | Accepted but inert on Sail: JARs have no classloader. The JVM overlay has a real one (verified) | 🔴 default / 🟠 JVM overlay |
| Concurrent Delta overwrite writers | Two independent Connect sessions race at one barrier; one commits and the other receives a transaction failure from the conditional Delta-log create | 🟢 probed conflict rejection |
| Fabric feature | Emulator | Type |
|---|
| SQL-analytics-endpoint semantics over lakehouse Delta | DuckDB runs real SQL (aggregation / join / filter), e2e | 🟢 Real (engine in e2e) |
| Warehouse item management | Full | 🟢 Real |
| T-SQL over TDS + Entra FedAuth | Pure-Go TDS front (internal/tds) terminates the FedAuth handshake (real Entra token, database.windows.net audience), then byte-splices the client’s post-login session to a real per-item SQL Server connection so the engine emits every token itself. Unmodified go-mssqldb and Microsoft ODBC Driver 18 (pyodbc) clients connect and run T-SQL — including RPCs, prepared statements, and transactions. Verified against a real SQL Server; Microsoft’s real dbt-fabric adapter passes debug/seed/run/test end-to-end (e2e/dbt-fabric/) | 🟢 Real (front + default sidecar) |
| Lakehouse SQL analytics endpoint — Delta → engine | The emulator reads the lakehouse’s Tables/<t> Delta in pure Go and reflects (CREATE+INSERT) it into the sidecar on connect, so SELECT hits real OneLake data (matches DuckDB), read-only (writes rejected). Not PolyBase — SQL Server reading Delta in place is a proven dead-end on the Linux container (a throwaway spike; see 16-warehouse-tds.md) | 🟢 Real (reflection) |
| Warehouse — read-write T-SQL | Client CREATE/INSERT/SELECT relay straight to the sidecar; the warehouse owns its data (no reflection) | 🟢 Real (relay) |
Fabric SQL Database (database/) — OLTP + OneLake mirror | Same read-write TDS/FedAuth path (its own SQL Server database), plus mirroring: POST …/sqlDatabases/{id}/refreshMirror snapshots every table to OneLake as Delta (real Parquet + _delta_log), so Spark / DuckDB / delta-rs query the operational data. Verified with a go-mssqldb-writes → mirror → Delta-reads-back e2e (gated). Continuous/CDC mirroring and write-back-to-Delta are the deferred edge | 🟢 Real (snapshot mirror) |
| Per-item isolation (each item = its own SQL Server database) | Lakehouse/Warehouse routed by type; per-item databases so they never collide | 🟢 Real |
| RBAC → SQL permissions | Workspace role enforced on connect: no role → rejected; Viewer → read-only; Contributor+ → read-write (warehouse) | 🟢 Real |
information_schema / sys.* introspection | Relays natively — reflected/warehouse tables are real SQL Server tables | 🟢 Real (relay) |
| Per-column type fidelity (real SQL types over the wire) | The splice forwards SQL Server’s own COLMETADATA, so every column carries its true native type over the wire (the re-encode fallback, used only by fake test backends, synthesizes INTN/FLTN/BITN and falls back to NVARCHAR text) | 🟢 Real (native) |
Nested CTEs (WITH x AS (WITH y AS …)) | Fabric supports standard, sequential and nested CTEs; the SQL Server sidecar rejects the nested form (Msg 156), which would make the emulator stricter than Fabric. The TDS layer closes the gap on the wire: it parses the WITH prefix with a real lexer (string literals, quoted identifiers, nestable block comments) and flattens the nesting into the sequential form the sidecar accepts, then re-encodes the batch. Statements Fabric itself refuses are rejected rather than rewritten — non-SELECT targets, DML or OPTION hints in a nested definition, same-level duplicate names, and references escaping their nesting scope (the last of these verified to otherwise succeed post-flattening, i.e. a real divergence, not a theoretical one). Cross-level name shadowing is refused by name rather than renamed, because rewriting references safely needs a parser this layer deliberately is not. Statements sent as parameters (sp_prepexec/sp_executesql) are rewritten too: the RPC parameter list is walked, the statement parameter re-encoded, and the result re-parsed and compared with the original before it is forwarded — every other parameter byte-identical, or the original goes instead. Unmodelled parameter types (TEXT/XML/UDT/SQL_VARIANT) forward untouched rather than being guessed at. Witnessed by examples/medallion-pyspark running dbt’s native accepted_values and relationships — which compile to nested CTEs — with no CTE-free substitute in the project, and by a parameterized-statement probe through the real ODBC driver (29-tsql-parity.md) | 🟢 Real (rewrite + refusals) |
CTAS (CREATE TABLE … AS SELECT) | Fabric/Synapse spell “materialise this query” as CTAS; SQL Server has no such statement and spells it SELECT … INTO, so the sidecar rejects valid Fabric T-SQL with Msg 156. The TDS layer rewrites it, splicing INTO before the first FROM at paren depth 0 — including inside the EXEC('…') dynamic SQL dbt-fabric actually ships, which a statement-level rewrite would miss. Fabric’s WITH (DISTRIBUTION = …) options are dropped (physical layout, not results). Witnessed by examples/medallion-pyspark building gold with dbt’s own +materialized: table (29-tsql-parity.md) | 🟢 Real (rewrite) |
Class B strict mode (-tsql-strict) | Refuses what real Fabric rejects but SQL Server accepts — recursive CTEs, triggers, synonyms, CREATE USER, SET TRANSACTION ISOLATION LEVEL/ROWCOUNT/IDENTITY_INSERT, FOR XML, IDENTITY(seed, increment), enforced key constraints, multi-column statistics, PREDICT, sp_showspaceused — so a locally green build means a Fabric-green build. Off by default: it removes capability, which is the operator’s call. Indexed views and FOR JSON-in-a-subquery stay unenforced, with the reason recorded | 🟢 Real (opt-in) |
| Connection by item name (vs GUID) | Workspace read from the server name (<workspace>.datawarehouse.fabric.microsoft.com), item resolved by display name; a GUID still resolves by id (back-compat). Verified with a real go-mssqldb client | 🟢 Real |
| Fabric feature | Emulator | Type |
|---|
Data Pipeline control flow (If / ForEach / Until / Switch / Filter / Fail, expression language, dependsOn) | Pure-Go interpreter that really executes | 🟢 Real (orchestration) |
| Per-activity policy — retry + backoff + timeout | Applied to every activity type: policy.retry re-runs a failed activity (each retry from scratch; only the final outcome is recorded, carrying retryAttempt); policy.retryIntervalInSeconds is folded into the run’s durationInSeconds as virtual backoff; policy.timeout fails an attempt whose own virtual duration exceeds the limit. No real sleeping — backoff and timeouts are exercised in milliseconds on the controllable clock | 🟢 Real |
ForEach sequential / parallel (isSequential, batchCount) | Iterations run in array order (deterministic); the mode sets the reported wall-clock — sequential iterations add, a parallel batch costs its slowest — matching how real Fabric overlaps them | 🟢 Real |
List pagination (continuationToken) | On by default, as in Fabric: a list past the server page size returns an opaque continuationToken and an absolute continuationUri, with no client opt-in — so a client that ignores the token breaks here exactly as it would in production. ?maxPageSize may narrow a page, never widen it. The page size is a testing lever (-list-page-size / FABRIC_LIST_PAGE_SIZE): set it to 2 and every list forces the token loop on the spot, the same idea as the controllable clock for LROs. Witnessed by Microsoft’s fab CLI — a plain list (no page-size parameter) comes back paged, and paging through is asserted complete: no item twice, none missed, terminal page tokenless | 🟢 Real |
| Invoke pipeline (ExecutePipeline) | Resolves the referenced DataPipeline (GUID or name, optional other workspace) and runs it through a fresh interpreter — real recursive interpretation, one level deeper on the same engines. waitOnCompletion (default) gates the parent on the child’s terminal status; parameters flow into the child; a cycle or excessive nesting fails loudly | 🟢 Real |
| Pipeline → notebook activity (TridentNotebook) | Resolves the notebook reference and creates a real RunNotebook job instance the pipeline gates on — the pipeline→jobs linkage is real; the notebook’s cells execute only on the Spark sidecar, and a job with cells outstanding has no clock-derived completion: it stays non-terminal until the engine reports, so a completed job means the cells ran | 🟢 Real chain / 🟡 cell execution witnessed at the binding layer only |
queryactivityruns detail | Full | 🟢 Real |
| Activity-level lineage | Successful Copy execution persists its resolved workspace/item/path source→sink edge, returns it in activity output, and exposes workspace lineage for OpenMetadata ingestion | 🟢 Real (Copy) |
| Copy activity — OneLake → OneLake | Really moves the bytes through the storage layer: a file, or a directory subtree preserving structure; source/sink locations {workspaceId?, itemId, path} are expression-resolved (GUID or name); returns real filesWritten / dataWritten. External stores / format transformation are out of scope and fail loudly | 🟢 Real (in-family) / 🔴 external |
| Lookup activity — OneLake CSV/JSON/Parquet/Delta | Reads real rows from a CSV, JSON, or standalone Parquet file, or a lakehouse Delta table (Tables/<name>, auto-detected — no format hint needed) in OneLake; honors firstRowOnly; the result flows into @activity(…).output for downstream steps. Parquet/Delta reuse the warehouse’s own Parquet reader — a real Delta column keeps its native type (int/float/bool), not a stringified cell | 🟢 Real (CSV/JSON/Parquet/Delta) |
| GetMetadata activity — OneLake path | Stats a real OneLake path: exists / itemType / size / lastModified / childItems; a missing path honestly returns exists:false | 🟢 Real |
| Script / SqlServerStoredProcedure activities | Run real T-SQL against a Warehouse/Fabric-SQL-Database item’s own SQL Server database — the same per-item backend the TDS endpoint and the SQLDatabase mirror share. Script runs each scripts[] entry (Query → real rows back, NonQuery → rows affected); SqlServerStoredProcedure calls a real stored procedure with named parameters. The target is named directly as {workspaceId?, itemId} (the emulator’s own scoped mapping — real Fabric’s linkedService/connection reference isn’t modeled), the same shape Copy/Lookup/GetMetadata already use. Honest error without a warehouse SQL backend attached | 🟢 Real (scoped) |
| Web / external-connector leaves | Stubbed success — reached in dependsOn order and inputs resolved, but nothing executes: Web calls to arbitrary URLs would break the offline/deterministic guarantee | 🟡 Emulated |
| Apache Airflow Job | Typed item + beta file APIs; uploaded Python DAGs sync to an opt-in real Airflow 2.10.5/Python 3.12 sidecar, whose scheduler/executor and REST state determine the Fabric job result (e2e/airflow) | 🟢 Real (sidecar) |
| Dataflow Gen2 (Power Query M engine) | Typed item/definition management round-trips. Refresh, Publish, and in-pipeline execution fail with DataflowEngineNotImplemented; no open Power Query M engine exists to attach | 🟡 mgmt / 🔴 exec |
| Connectors / on-prem gateways | — | 🔴 Not implemented |
| Fabric feature | Emulator | Type |
|---|
| Git integration (connect / status / commit / update / disconnect) | Full, real state | 🟢 Real |
fabric-cicd tool publishing | The real client round-trips definitions (e2e) | 🟢 Real |
| Deployment pipelines — model, assignment, item pairing, Deploy Stage Content (D0–D2) | Real promotion: definitions really copy, pairs decide (not names), metadata only — a deployed lakehouse arrives empty — and target-only items survive. 202 LRO + /result detail. 23-deployment-pipelines.md | 🟢 Real |
| Deployment pipelines — role-assignment CRUD (D3) | Add / Delete / List; Admin is the only role a pipeline defines; mutations require Admin, reads require membership | 🟢 Real |
| Fabric area | Emulator | Type |
|---|
Real-Time Intelligence — Eventhouse / KQL Database (real-time-intelligence/) | Full item management (including the default child database an eventhouse creates, and creationPayload.parentEventhouseItemId), plus the Kusto REST protocol on the eventhouse’s published properties.queryServiceUri — /v1/rest/mgmt, /v1/rest/query, /v2/rest/query — terminated by the emulator (Kusto-audience bearer, workspace RBAC, one isolated engine database per Fabric KQL Database) and executed by Microsoft’s own KQL engine container (kustainer) when the rti profile attaches it. No engine attached → honest 501. 25-rti-kusto.md | 🟡 mgmt / 🟠 exec (opt-in --profile rti) |
Real-Time Intelligence — Eventstream (real-time-intelligence/event-streams/) | Item management only. The attached Kusto engine is a query/ingest engine with no streaming ingestion — a streaming pipeline is a different service, deferred with cause | 🟡 mgmt / 🔴 exec |
CopyJob, KQLDashboard, KQLQueryset, WarehouseSnapshot | Typed collections over the generic item surface — create/get/list/patch/delete and definition round-trip, with the collection forcing its type. Type names taken from fabric-docs payloads; no execution engine for any of them (a Copy Job does not copy) | 🟡 mgmt / 🔴 exec |
Event triggers — Reflex (Data Activator) on OneLake events | The Reflex triggers. A trigger subscribes to a source item’s OneLake file events (FileCreated / FileDeleted / FileRenamed) under an optional path prefix, and a match starts a real item job — invokeType: "EventTriggered" — with the event bound in as @pipeline()?.TriggerEvent?.FileName / .FolderPath / .Subject, reading which needed safe navigation (?.) in the expression language, so one definition works whether or not a trigger started it. No broker is needed: every byte written to OneLake passes through the emulator’s own storage layer, so a file event is observable at the source whoever wrote it — an ADLS client, azcopy, delta-rs, a Copy activity, the mirror writer. Dispatch is synchronous and reentrant, so bronze→silver→gold chains work; a runaway is bounded by an activation count rather than by identity, matching how Fabric bounds one — Activator documents Fabric item — Activations/user/minute — 50 and throttles or cancels beyond it, and caps input at 10,000 events/second/rule, with no documented loop detection or dedup. Independent events each activate, because Activator “continues monitoring without waiting for the action to complete”. Boundary: the binding is an emulator-native control surface (…/reflexes/{id}/triggers) because Fabric has no public REST for it — the Eventstream/Reflex rule is assembled in the portal. What is faithful is everything downstream: the filter, the invocation, the TriggerEvent fields, and a real pipeline really running. 07-control-plane-api.md | 🟢 Real (OneLake sources) |
Mirroring — Mirrored Database (mirroring/) | POST …/mirroredDatabases/{id}/refreshMirror mirrors an external SQL Server source (reached via a Connection with Basic credentials) to OneLake as real Delta — reusing the exact same mirror writer the Fabric SQL Database uses (warehouse.Mirror; same code, external source). Proven by a gated e2e: a table seeded directly on an external database (bypassing the emulator’s own per-item routing entirely) mirrors and reads back correctly. Snapshot-on-trigger, not continuous/CDC replication; other source engines (Snowflake, CosmosDB, on-prem via gateway) are out of scope | 🟢 Real (snapshot mirror, SQL Server sources) |
Power BI — Semantic Model query (executeQueries) | Real bounded DAX engine — EVALUATE, SUMMARIZECOLUMNS, measures, SUM/DIVIDE, relationship filter propagation — over imported data.json or compatibility-level-1604 Direct Lake entity partitions backed by current OneLake Delta. Proven by the golden DAX/GX suites and the Spark-written Direct Lake witness. | 🟢 Real (DAX subset + Direct Lake) |
| Power BI — Reports / rendering; full DAX; SemPy over XMLA | No report rendering; DAX beyond the fixture subset; and the native ADOMD.NET/XMLA transport SemPy uses (no CI oracle) — all deferred with cause | 🟡 mgmt / 🔴 render |
Data Science — ML models / experiments / MLflow (data-science/) | Authenticated, workspace-scoped proxy to a real MLflow 3 tracking/model-registry server. Experiment/model creation synchronizes typed Fabric items; experiment/run references are isolated by workspace; successful artifact uploads are mirrored under the experiment item’s OneLake Files/mlflow-artifacts. | 🟢 Real (sidecar) |
Graph (graph/), Real-Time Hub, Copilot / IQ (iq/), Embed, Workload Dev Kit | — | 🔴 Not implemented |
| Capability | Purpose |
|---|
Controllable clock (/_emulator/clock) | Advance virtual time to drive LRO / job status transitions deterministically. |
Fault injection (/_emulator/faults, /_emulator/permissions) | Force failures / throttling / RBAC denials to test client resilience. |
Flow stream (/_emulator/events) | Server-Sent Events for every byte that moves through OneLake — file events for writes/renames/deletes whoever made them, and table events derived from Delta commits (version, rowsAdded, filesAdded). Plus job and activity events — a failing activity reaches the stream with its error and job id the moment it fails, rather than being reconstructed from queryactivityruns afterwards — and attribution: a Copy activity’s writes name the activity, a notebook cell’s name the cell (including engines that cannot set headers, via the same bearer claims lineage uses). Never inferred; a write with nothing to say carries no attribution. curl -N tails a medallion run live; ?kinds= narrows it and ?since= replays the ring buffer. Delivery is deliberately lossy — a slow consumer is told what it dropped rather than being allowed to stall a writer. Two further kinds complete a medallion’s chain: lineage when a movement is recorded (so a graph redraws as a warehouse build happens, which has no job to end), and query when a semantic model is read — the Power BI hop, which is an event and never an edge because a query moves no data. 31-flow-observability.md |
| Data-flow lineage beyond OneLake | A medallion’s last hops move no bytes through OneLake, so each is observed on its own terms rather than inferred. silver → gold: the TDS front already parses every statement for dialect adaptation, so it also records what the engine accepted — CTAS, SELECT … INTO, INSERT … SELECT, CREATE VIEW, sp_rename, DROP, including inside the EXEC('…') dbt-fabric actually ships — resolving three-part names to Fabric items (producer: Warehouse). dbt’s build-then-swap is followed through sp_rename, so the graph names fct_order_lines and not its __dbt_temp scaffold. A statement whose response carries an error, or whose outcome cannot be read, records nothing. gold → semantic model: a Direct Lake table’s binding names its source, so the edge is recorded when the definition lands (producer: DirectLake); an import model gets none, because its rows arrive detached from wherever they were selected and inventing a source would be a guess. Interactive engines: POST /v1/workspaces/{wid}/lineage is the notebook read/write report without the job, for a Spark session or plain script (producer: Reported) — emulator-native, as Fabric has no such endpoint. The producer is the point: Warehouse is evidence, Reported is a claim, and a consumer never has to guess which. 31-flow-observability.md |
| Svelte management portal | Dashboard, workspaces, operations, clock, and fault controls. |
Parity isn’t claimed from our own tests alone — each 🟢 surface is pinned against
the real, unmodified client a Fabric user runs, executed against the emulator
in CI (e2e/<client>/). If Microsoft’s own tool round-trips unchanged, the
contract holds better than any assertion we could write ourselves.
| Real client (pinned) | Surface exercised | Status |
|---|
fabric-cicd (Microsoft) | Control plane / CI-CD publish | 🟢 e2e/fabric-cicd |
Fabric CLI fab (Microsoft) | Control plane — SPN auth (MSAL) + workspace/item CRUD (Notebook, SemanticModel, Report, DataPipeline, Lakehouse), ls/get/api | 🟢 e2e/fabric-cli |
| Fabric Data Engineering VS Code extension 1.18.1 contract (Microsoft) | Shared-backend/MWC authoring routes through api.powerbi.com; interactive kernel websocket is not claimed | 🟢 e2e/vscode-extension |
| Apache Airflow 2.10.5 | ApacheAirflowJob DAG discovery, scheduling, execution, and status | 🟢 e2e/airflow |
| MLflow 3 + dbt-duckdb | Workspace-scoped experiment/run/artifact/model lifecycle, followed by dbt’s real Delta plugin over the same Spark-written OneLake table | 🟢 e2e/data-science-loop |
deltalake (delta-rs) | OneLake Delta write/read | 🟢 e2e/delta-rs |
azure-storage-file-datalake + Blob SDK | OneLake ADLS Gen2 DFS + Blob | 🟢 e2e/adls-sdk |
azcopy (Microsoft) | OneLake Blob multi-block transfer | 🟢 e2e/azcopy |
| DuckDB | Lakehouse SQL over Delta/Parquet | 🟢 e2e/duckdb |
| PySpark behind the Livy API | Spark sessions / statements | 🟢 e2e/spark, e2e/livy, e2e/notebook-run |
notebookutils | Notebook utility shim | 🟢 e2e/notebookutils |
go-mssqldb | Warehouse/Lakehouse TDS + FedAuth | 🟢 internal/server, internal/tds |
dbt-fabricspark (Microsoft) | Fabric Spark via Livy HC sessions | 🟢 e2e/dbt-fabricspark — debug→seed→run→test on Sail |
dbt-fabric (Microsoft) | Warehouse TDS via ODBC Driver 18 | 🟢 e2e/dbt-fabric — debug→seed→run→test through the TDS splice |
azure-kusto-data (Microsoft) + raw Kusto REST, over kustainer (Microsoft’s own KQL engine) | Eventhouse / KQL Database: /v1/rest/mgmt, /v1/rest/query, /v2/rest/query on the published queryServiceUri — create table, ingest, query values back, per-database isolation | 🟠 e2e/rti — witness of record is CI (amd64). The engine needs AVX2, which Rosetta does not provide, so the default Docker setup on Apple silicon cannot run it; a QEMU x86-64 VM with --cpu-type max can, and does (25-rti-kusto.md) |
The TDS surface now has two independent driver witnesses: go-mssqldb and
the Microsoft ODBC Driver 18 (via dbt-fabric). That second driver mattered —
it exposed a real gap: go-mssqldb tolerated a synthesized FedAuth login, but
ODBC Driver 18 took a compatibility path (prepared-statement RPCs +
sp_reset_connection under mandatory connection pooling) that desynced against a
re-encoding relay. The fix was to byte-splice the post-login session straight
to the real SQL Server (so it emits every token itself), which is exactly the
kind of driver-family gap a single-driver test never surfaces. dbt-fabricspark
likewise drives the high-concurrency Livy layer over its real Livy-session
protocol (method: livy, service-principal auth via entra-emulator).
The emulator targets Microsoft Fabric — the convergence/successor product —
not the earlier Azure analytics services Fabric replaced. That boundary is why
some adjacent dbt adapters and Azure surfaces are intentionally not built:
they belong to predecessor (often retired) products, and their Fabric-native
successors are what we emulate instead.
| Adjacent product / client | Why out of scope | Fabric-era equivalent (in scope) |
|---|
Azure Synapse dedicated SQL pool (dbt-synapse) | Different product: its own control plane (Synapse workspaces) and an MPP T-SQL dialect (DISTRIBUTION = HASH, clustered-columnstore / resource-class DDL) that our vanilla SQL Server sidecar rejects. dbt-synapse layers on dbt-fabric, so the shared SQL path is already covered by the dbt-fabric witness | Fabric Warehouse — 🟢 TDS relay |
Azure Data Lake Analytics — U-SQL / SCOPE (dbt-scope) | Retired service (EOL Feb 2024), proprietary batch language, no Fabric embodiment. The only overlap (Delta on a lake) is Spark/OneLake, already witnessed | Fabric Spark — 🟠 Livy |
| ADLS Gen1 | Retired (Feb 2024), superseded by Gen2 | — |
| ADLS Gen2 (standalone storage account) | Not missing — OneLake is the Gen2 endpoint: hierarchical namespace, the dfs filesystem API, onelake.dfs.fabric.microsoft.com. Fabric has no separate storage account to emulate | OneLake — 🟢 e2e/adls-sdk |
Rule of thumb: if a capability exists only in a product Fabric replaced, it’s out
of scope; its Fabric-native successor is what we build. “We already have the
TDS/SQL Server foundation” makes Synapse cheaper, not done — the remaining
delta is a whole MPP dialect plus a second control plane, for a superseded
target. So the two dbt adapters we build (dbt-fabricspark, dbt-fabric) are
exactly the two that hit live Fabric surfaces; the other two (dbt-synapse,
dbt-scope) target predecessor products outside the emulator’s remit.
Real Fabric’s own Livy endpoint is Microsoft’s implementation of the Livy REST
contract over their Spark platform — they honor the protocol, not the retired
Apache Livy server. And where Fabric adds its own layer on top of that
protocol — high-concurrency REPL packing, which a vanilla Livy server has no
concept of — the emulator implements that layer directly rather than proxying,
because there is nothing to proxy it to. That is the same stance throughout: the
protocol and control plane are the durable, real things (built, not mocked,
so real clients can’t tell the difference), and the compute engine is attached
(Spark; T-SQL on SQL Server; KQL on Microsoft’s own Kusto engine —
25-rti-kusto.md) or deferred when proprietary or without an
implementation to attach at all (Dataflow Gen2’s M engine, Power BI rendering,
Eventstream’s streaming ingestion). Every deferral fails loudly rather than
pretending to succeed. See 13-roadmap.md for the milestone
history and the deferred-with-cause rationale.