Parity — v0.20.0
How the emulator’s surface maps to real Fabric (as documented at learn.microsoft.com/fabric / MicrosoftDocs/fabric-docs), and — the point of this table — whether real work happens or just the API shape.
The emulator’s design bet is that the durable, testable surface is
contracts + storage + identity + orchestration, and those are done for real
(real signed JWTs, real Delta bytes on disk, real RBAC, a real pipeline
interpreter, real cross-engine SQL, real Livy high-concurrency session packing).
The heavyweight or proprietary compute engines run by default: docker compose up auto-loads an override that starts Sail (Spark Connect) and a SQL
Server sidecar, so Livy sessions, notebook cells and the T-SQL warehouse do real
work out of the box. What stays 🟠 is the narrower set that needs a different
engine — the JVM overlay, or an opt-in profile — and what cannot be done
honestly at all is stubbed.
“Real via our own wire-protocol implementation.” A row is 🟢 Real not only when an external engine/client does the work, but also when the emulator itself implements Fabric’s wire protocol and the logic behind it — so a real, unmodified client gets byte- and behaviour-identical responses. Fabric’s control plane, OneLake’s ADLS/Blob surfaces, the Data Pipeline expression language + control flow, and the Livy high-concurrency session-packing layer are all in this category: no engine is being proxied, yet the observable contract matches real Fabric because we built the protocol, not a mock of it. Where a row’s execution needs an engine that is not the default — JVM-only Spark surfaces, an opt-in KQL profile — that part is split out as 🟠 or 🔴.
Legend
Section titled “Legend”| Meaning | |
|---|---|
| 🟢 Real | Genuine work: real signed JWTs, real bytes on disk, a real engine/client computes, real logic enforced — no pretending. |
| 🟡 Emulated | Faithful API contract + persisted state, but no engine — status is clock-derived / management-only. |
| 🟠 Non-default engine | Real, but only on an engine that is not the default: the JVM Spark overlay (docker-compose.spark-jvm.yml) or an opt-in profile such as --profile rti. Not “bring your own”: docker compose up already starts Sail and the SQL Server sidecar, so the everyday Spark and T-SQL surfaces are 🟢. |
| 🔴 Not implemented | Honest 501 or absent. |
Platform / fundamentals
Section titled “Platform / fundamentals”| Fabric feature | Emulator | Type |
|---|---|---|
| Workspaces CRUD | Full. Display names are unique tenant-wide — duplicates 409 WorkspaceNameAlreadyExists (uniqueness per the REST reference; fabric-docs covers workspace naming portal-side only) | 🟢 Real |
| Items CRUD + typed collections | Full. Display names are unique per (workspace, type) — duplicates 409 ItemDisplayNameAlreadyInUse; names stay reusable across types, which is why OneLake addresses items as name.Type. The type is validated against the documented ItemType enumeration (50 values, REST reference) — anything else is InvalidItemType, as real Fabric returns — and canonicalised case-insensitively so notebook and Notebook cannot become two types. 24 typed collections alias the generic surface; each collection segment is taken from its own reference page, since they are not derivable (GraphQLApis is capitalised, variableLibraries is not) | 🟢 Real |
| Role assignments / workspace RBAC | Enforced from the validated bearer principal | 🟢 Real |
| Folders | Full | 🟢 Real |
| Capacities (list, assign / unassign) | Full state, no billing/SKU enforcement | 🟢 Real state |
| Long-running operations (202 → poll) | Clock-derived | 🟡 Emulated |
Item job execution (jobs/instances) | Five item types really execute, and their status follows the work rather than the clock. Each parks CompleteAt beyond the clock’s reach at submit — so no poll can observe a clock-completed job whose work is still running — and finalises when the work reports: DataPipeline runs the interpreter, CopyJob moves the bytes, Notebook and SparkJobDefinition are executed by the Spark agent when one is configured (without an agent they stay open for the documented external callback, which is the honest contract rather than a hang), and Apache Airflow Job is decided by a real Airflow sidecar. An item type with no execution engine — a Lakehouse, Warehouse, SemanticModel, Reflex and most of the ~50-type enumeration — still accepts the POST, because Fabric’s job surface is generic across item types, and its status is then derived from the virtual clock (NotStarted → InProgress → Completed as CompleteAt passes) rather than from work. The lifecycle is faithfully shaped; nothing ran. That is the honest grade for a type with nothing to execute — inventing a work-derived status there would be the lie the notebook reconciliation was built to kill. Dataflow’s refusal is the one outcome still reached at submit, because a 501 really is instantaneous | 🟢 Real (types with an engine) / 🟡 clock-derived where there is none |
List Item Job Instances (GET …/jobs/instances) | An item’s runs, paged, newest first, with the same clock-derived status the single-instance read returns. Listing is itself an evaluation point for the scheduler below, so a run that has come due appears without touching the control surface | 🟢 Real |
Item Job Scheduler (…/items/{id}/jobs/{jobType}/schedules) | Fabric’s own per-item scheduler — distinct from the ApacheAirflowJob item, which delegates to a real Airflow sidecar. All four ScheduleConfig members: Cron (interval 1–5,270,400 min), Daily/Weekly (times[] ≤ 100, weekdays[]), Monthly (DayOfMonth or ordinal weekday, recurrence 1–12) — every documented bound enforced at write time, the 20-per-item ceiling as ScheduleExceedsLimit, and localTimeZoneId (Windows ids or IANA) honoured as real local wall time, so a daily 09:00 stays 09:00 across a DST change. Schedules really fire: each due occurrence starts a job instance down the same path a manual run takes, so a scheduled DataPipeline executes the interpreter and only invokeType: "Scheduled" tells them apart. Evaluation is driven by the controllable clock rather than a background worker — advancing time materialises exactly the occurrences that came due, and a past startDateTime triggers instantly with no special case. Boundary: catch-up is capped at 100 occurrences per evaluation (newest kept), because a controllable clock can be advanced a year against a one-minute Cron. 07-control-plane-api.md | 🟢 Real |
Identity & security (security/, admin/)
Section titled “Identity & security (security/, admin/)”| Fabric feature | Emulator | Type |
|---|---|---|
| Entra OAuth2 tokens / JWKS / client-credentials | entra-emulator mints real signed JWTs | 🟢 Real |
| Workspace managed identity handshake | Provisioned via entra admin API; the identity’s own token passes RBAC | 🟢 Real |
| Key Vault references in connections | Resolved against azure-keyvault-emulator | 🟢 Real |
Governance domains (/v1/admin/domains) | Full admin surface: domain/subdomain CRUD (the hierarchy stops at two levels, as documented), workspace assignment (single-valued — re-assigning moves a workspace), nonEmptyOnly listing, and bulk Admins/Contributors role assignment. Deleting a domain cascades to its subdomains, assignments and roles. Tenant-admin gated, and graded as Microsoft documents it rather than uniformly: a READ (admin/workspaces, admin/items) requires “a Fabric administrator or a service principal”, while a WRITE (admin/domains/create-domain and every other mutation) requires “a Fabric administrator” with no service-principal escape — so a service principal may read the tenant and may not change it. Administrators are declared by the operator (FABRIC_TENANT_ADMINS), never inferred from a token claim; with none declared every mutation is refused, because the pre-gate behaviour was that everyone was an admin. Refusals are InsufficientPrivileges/403, the documented code | 🟢 Real (mgmt + tenant-admin gate) |
Activity log / audit (GET /v1.0/myorg/admin/activityevents) | Real audit trail — the emulator records events as operations happen, using the documented audit vocabulary (CreateWorkspace, CreateArtifact/UpdateArtifact/DeleteArtifact from admin/operation-list.md; InsertDataDomainAsAdmin & co. with their DataDomainObjectId/FoldersToSetCounter properties from governance/domains-audit-schema.md). Enforces the documented request rules (single-quoted UTC bounds, same-day window) and pages with continuationToken/continuationUri until the token stops coming back. Nothing is synthesised at read time | 🟢 Real |
Tenant settings (GET /v1/admin/tenantsettings) | The documented TenantSetting object in full — settingName, title, enabled, canSpecifySecurityGroups, tenantSettingGroup, the three delegateTo* flags, enabled/excluded security groups (graphId/name), and typed properties validated against the documented TenantSettingPropertyType enum. Optional arrays are omitted, not null, as the reference’s sample does; seeded with the setting names that sample uses. POST /v1/admin/tenantsettings/{name}/update is the real update API (enabled required; response wrapped as {"tenantSettings":[…]}) | 🟢 Real |
Tenant-wide workspace admin (GET /v1/admin/workspaces) | The documented Workspace shape — note it differs from the user-facing surface: the envelope key is workspaces (not value) and the field is name (not displayName). Filters type/state/capacityId/name are enforced, with undocumented enum values returning BadRequest as the reference specifies; domainId is reported from the real domain assignment. Emulator workspaces are always Workspace/Active (no soft delete), so state=Deleted is legitimately empty. Tenant-admin gated on the read rule — an administrator or a service principal, as this API’s reference states | 🟢 Real (mgmt + tenant-admin gate) |
Tenant-wide item admin (GET /v1/admin/items) | The documented Item shape across every workspace, with workspaceId/capacityId/type/state filters and the reference’s own error codes (InvalidItemType, InvalidItemState). Envelope key is itemEntities — a third spelling after the user-facing value and admin workspaces’ workspaces, which is why each came from its own reference page. Active is the only documented state, so that is all the emulator reports. Fields it does not model (creatorPrincipal, defaultIdentity, tags) are omitted rather than faked | 🟢 Real (mgmt + tenant-admin gate) |
Capacity tenant-setting overrides (GET /v1/admin/capacities/delegatedTenantSettingOverrides, POST …/{capacityId}/delegatedTenantSettingOverrides/{name}/update) | The documented CapacityTenantSetting (a tenant setting plus delegatedFrom and delegateToWorkspace, and without delegateToCapacity/delegateToDomain). The update body is the documented one — enabled required, no properties, no delegatedFrom — and the response wraps as {"overrides":[…]}. An override may only be created for a setting whose delegateToCapacity is true. Note there is no /v1/admin/capacities list API in Fabric: capacities are listed on the Core surface at /v1/capacities | 🟢 Real (mgmt + tenant-admin gate) |
Sensitivity labels (bulkSetLabels / bulkRemoveLabels) | The documented admin bulk APIs, reporting per-item successfulItems/failedItems rather than failing whole calls. Every change writes the documented SensitivityLabelEventData to the audit log — SensitivityLabelApplied/Changed/Removed with SensitivityLabelId, OldSensitivityLabelId, ActionSource 3 (Manual), ActionSourceDetail 5 (PublicAPI), ArtifactType 12, and a LabelEventType genuinely computed from label order (upgraded/downgraded/same-order/removed). The label taxonomy is emulator-provided — real Fabric gets labels and their order from Purview, which cannot be attached offline; GET /v1/admin/labels exposes it. Gated as writes (administrator only), but note what the gate does not model: these two APIs are documented User: Yes / Service principal: No, so real Fabric refuses a service principal even when it is an administrator. The gate models the administrator requirement, not Microsoft’s per-API Entra identity-support matrix | 🟢 Real (label APIs + audit) / 🟡 taxonomy |
| Sensitivity labels → catalog | Labels are Purview Information Protection objects, not Atlas entities, so the Atlas-API route a Purview → OpenMetadata migration takes cannot carry them — assets, classifications, glossary and lineage cross; labels do not. The optional OpenMetadata profile exports them instead: the taxonomy becomes classification FabricSensitivity (mutuallyExclusive, since an item carries at most one label), each label a tag under it, and an item’s sensitivityLabel is applied to that item’s catalog entity. CI reads the tag back through OpenMetadata’s API, on two items with different labels, and asserts that clearing the label in Fabric clears the tag in OM (22-openmetadata.md) | 🟢 Real (stored in OM) |
Purview Data Map ({endpoint}/datamap/api/atlas/v2/…) — split out of the old single “Purview scanning / classification” row, because its three parts have different ceilings: this one is reachable and moving, scanning is reachable in-family, and the built-in classifiers can never move. One 🔴 made them look equally stuck, and became actively wrong the moment this landed | The Data Map is Apache Atlas v2 — the Azure spec’s own TypeSpec declares @route("/atlas/v2/entity"), /types, /relationship, /glossary, /lineage and annotates them “This is Atlas API” — so this is Atlas semantics, not a Purview-shaped façade. Implemented: the type system (typedef CRUD, per-category reads, typedefs/headers) and entities (createOrUpdate, read/delete by GUID, /entity/bulk, lookup by unique attribute), on the documented purview.azure.net audience with the documented AtlasErrorResponse codes. What is enforced rather than shaped: an entity naming an unregistered type is refused; required attributes are checked through the supertype chain, which is how qualifiedName — declared once on Referenceable — is required of every type; qualifiedName is the per-type identity, so createOrUpdate genuinely updates instead of duplicating; Atlas’s negative-GUID placeholder protocol is honoured and reported in guidAssignments; a batch validates fully before it writes anything; deletes are soft, as the spec’s EntityStatus states. The four Atlas base types are seeded, as a real account has them. Scope, explicitly: 96 routes in the spec, and this is the type system and entities only — glossary, lineage, relationships, classifications, business metadata and search are NOT implemented. 🟡 rather than 🟢 for one reason: the row’s evidence is 14 Go tests, and this repo’s rule is that green needs a real-client witness in CI. pyapacheatlas and Microsoft’s azure-purview-datamap both speak exactly these routes, so that witness is available and is the next increment — until it runs, the grade stays 🟡 | 🟡 Emulated (real type system + entities; no third-party witness yet) |
Purview scanning (scan/) | Not implemented. A scan is an engine, not a surface: it connects to a source, enumerates assets, infers schema and applies rules over real values. Against the emulator’s own sources — OneLake Delta, the warehouse — that is genuinely computable, so this row has a reachable ceiling of 🟢 in-family / 🔴 external, which is why it is not bundled with the row below | 🔴 Not implemented |
| Purview system classifiers (the ~200 built-ins) | Not implemented, and this one can never move. The built-in classifiers are proprietary detection patterns published in no OpenAPI spec and in no documentation; custom classification rules are implementable, the shipped set is not. Split out from scanning precisely because its ceiling is different: scanning is blocked by work, this is blocked by information nobody outside Microsoft has | 🔴 Not implemented (no reachable ceiling) |
| Lineage (catalog graph) | Via the optional OpenMetadata profile: OneLake shortcut edges and executed pipeline Copy source→sink edges are persisted exactly and witnessed in OM’s graph API. Notebook edges come from the engine’s own report of what it read and wrote, or from the data plane observing it; Script/stored-procedure code is still not guessed. Catalog SSO can also be pointed at entra-emulator (22-openmetadata.md) | 🟢 Real (shortcuts + Copy) |
OneLake (onelake/)
Section titled “OneLake (onelake/)”| Fabric feature | Emulator | Type |
|---|---|---|
| ADLS Gen2 DFS surface (create → append → flush, ranged read, list) | Full, incl. the x-ms-range dialect | 🟢 Real (real bytes) |
| Blob surface | Full | 🟢 Real |
| Delta commits (put-if-absent atomicity) | Real; -race-tested concurrent-commit race | 🟢 Real |
| Shortcuts (OneLake → OneLake) | Symlinks with target-side RBAC (trusted-workspace-access) | 🟢 Real |
| Shortcuts to external targets (S3 / ADLS Gen2 / Dataverse) | Real Amazon S3 / S3-compatible read-through: a Connection carrying the Access Key Id + Secret Access Key pair — as Basic credentials, because Fabric’s S3 connector uses authentication kind “Access Key” while the REST reference’s CredentialType enum has no AccessKey member and Basic is its only two-secret type — makes the emulator sign upstream requests with AWS SigV4 (internal/awssig, verified against AWS’s own published example signature). Witnessed by e2e/s3 against a real SeaweedFS server started with an identity config — the suite proves an unsigned GET is 403 and that a wrong secret is refused, so the pass cannot be vacuous; the object itself is written by boto3. ADLS Gen2 reads are witnessed too: e2e/azurite-shortcut drives a real SAS through the shortcut against Azurite, Microsoft’s own storage emulator, proving an unauthenticated GET is 403 and a tampered SAS is refused. Scope stated honestly — Azurite implements Blob/Queue/Table and not ADLS Gen2, so the endpoint is the Blob one; that witnesses the read path (a plain authenticated GET, identical on both endpoints of a real ADLS Gen2 account) but not DFS-specific behaviour, which stays unwitnessed offline. Dataverse is now a real target type, at a scope worth stating precisely: Microsoft documents the target’s four fields (connectionId, deltaLakeFolder, environmentDomain, tableName — the only documented target with no location) and documents that the tables must already exist in the Dataverse Managed Lake. It does not document that lake’s byte layout, so the emulator serves ordinary Delta from environmentDomain/deltaLakeFolder/tableName and that composition is ours, not Microsoft’s. This is a target type in the existing shortcut machinery, not a Dataverse emulator: no Dataverse Web API, no OData metadata, no table discovery. The read-only rule is real and enforced: Fabric states twice that “Dataverse shortcuts are read-only. They don’t support write operations regardless of the user’s permissions”, and both the flush and the delete paths refuse, with e2e/azurite-shortcut confirming via the Azure SDK that the refused write never reached the target and the refused delete left it intact. Auth diverges and is disclosed: Fabric’s delegated model is organizational account (OAuth2) or service principal, while the emulator presents whatever the Connection carries — the witness uses a SAS, so what is proven is that the credential is really presented and really checked, not that OAuth2 delegation works. S3 read-only is by design — Fabric documents S3 shortcuts as read-only regardless of permissions, so a write there is refused locally and never reaches the target. ADLS Gen2 write-through is real: a file flushed under an ADLS shortcut is PUT to the storage account and the local buffer dropped (it would otherwise shadow later target changes), and deleting a file within a shortcut deletes it at the target, as documented. Witnessed by e2e/azurite-shortcut, which reads the result back with the Azure SDK straight from Azurite rather than trusting the emulator’s own 200 | 🟢 Real (S3 SigV4 + ADLS reads) / 🟠 Dataverse (opt-in layout) |
Data Engineering (data-engineering/)
Section titled “Data Engineering (data-engineering/)”Engine-level claims below are measured, not asserted: the generated Spark engine matrix probes each capability against both Sail and the JVM overlay and is regenerated in CI.
| Fabric feature | Emulator | Type |
|---|---|---|
| Lakehouse item + Tables/Files storage | Full (via OneLake) | 🟢 Real |
| Notebook authoring / definition round-trip | Full | 🟢 Real |
notebookutils / mssparkutils (fs, credentials, getSecret, lakehouse, runtime) | Functional stdlib shim (python/notebookutils) | 🟢 Real |
notebookutils.notebook.runMultiple — DAG structure and ordering | Runs a DAG in dependency order, accepting a bare list or Fabric’s {activities, timeoutInSeconds, concurrency} shape. A failed activity’s dependents are reported skipped with the reason rather than run. A cycle, a dependency naming an activity outside the DAG, a duplicate activity name, or a malformed shape is refused. validateDAG exposes those same checks ahead of a run, and workspace accepts a name or an id | 🟢 Real |
notebookutils.notebook.run — exit values | Returns the child’s exit value, the exact string passed to notebookutils.notebook.exit(...), and "" when the child never called it. Applies to runMultiple’s per-activity exitVal too. Until 0.18.0 this returned the terminal job status, so a parent branching on a child’s exit value took one path here and the other on Fabric | 🟢 Real |
runMultiple — failure contract | Raises RunMultipleFailedException (importable from notebookutils.common.exceptions) when any activity did not complete, carrying every result on .result. Each value carries Fabric’s exitVal and exception keys; status/error ride alongside as emulator extras for local debugging and nothing should depend on them | 🟢 Real |
| Materialized lake views | The refresh is real: the view’s own query runs on the engine the emulator already hosts and the result is written as a Delta table under Tables/<name>, so a Delta reader, the SQL analytics endpoint, the flow stream and lineage all see a refresh exactly as they see any other write. A refresh is only reported successful if a Delta commit actually landed — checked against OneLake rather than inferred from the statement’s exit code, because a cheerful statement that wrote nothing is precisely how a view whose rows do not exist would be reported Materialized. Staleness is measured, not assumed: the declared sources’ Delta versions are recorded at refresh time (read before the query, so a write landing mid-refresh cannot be counted as data the view read — forced in the witness by advancing a source from inside the engine fake) and compared with what those tables are now; the answer names which source moved. A view that has never refreshed is NeverRefreshed, not stale — there is nothing there to be out of date. Emulator-native definition surface, and deliberately: Fabric defines these with Spark SQL DDL that no capture here has observed, and this repo does not invent a syntax — so the definition hangs off …/lakehouses/{id}/materializedlakeviews, on the same precedent as the Reflex trigger binding, and dependsOn is declared rather than parsed out of the SQL (a wrong parse would not fail, it would silently report a stale view as fresh). Dropping a definition leaves the materialised table: the rows are real data a reader may still be using. When a DDL capture arrives it becomes a second front door onto this model, not a rewrite | 🟢 Real (refresh + staleness) / 🟠 definition surface emulator-native |
| Reference-run lakehouse rule | A referenced child notebook bound to a different default lakehouse than its parent is blocked, matching Fabric; a child that declares none inherits, a matching one runs, and useRootDefaultLakehouse in the arguments bypasses the check. The failure names the cause and the way out rather than “The job failed.” | 🟢 Real |
runMultiple — retry and timeouts | Per-activity retry/retryIntervalInSeconds (last error reported, no trailing wait after a final attempt) and the DAG-level timeoutInSeconds, defaulting to Fabric’s 12 hours. timeoutPerCellInSeconds is applied per cell, multiplied by the notebook’s real cell count | 🟢 Real |
runMultiple — concurrency | An explicit concurrency is honoured with a bounded pool per dependency level, 0 meaning unlimited; a level boundary stays a barrier and results keep their listed order regardless of completion order. The default diverges deliberately: Fabric defaults to 3× the CPU count, the emulator to sequential, because a harness comparing two runs needs the same sequence more than it needs speed | 🟡 Real, sequential by default |
runMultiple — child Spark sessions | Fabric runs children on isolated REPL instances within the parent’s Spark session, so they share its compute and its session-scoped state. The emulator gives each child its own session, so sibling activities cannot see one another’s temp views. Deliberate: the benefit is narrow and the change is large. Scoped in 39-run-multiple-parity-plan.md | 🟡 Isolated sessions |
| Spark session / statement / batch via the Livy API | Native termination (--spark-agent-url): the emulator implements the Livy contract and drives a persistent statement-executor agent. The default agent is a PySpark Connect client of Sail, not Apache Spark. An external Livy backend remains configurable with --spark-livy-url | 🟢 Real (default engine) |
| Notebook cell execution | The emulator parses and records the notebook run, resolves attached lakehouse/Environment metadata, and Sail executes cells by default. The same fixture runs on Spark 3.5 JVM and proves unqualified saveAsTable/spark.table bind to OneLake Tables/ | 🟢 Real (default engine; per-capability detail in engine-matrix.md) |
| Livy High-Concurrency (5-REPL) sessions | Fabric’s packing layer is implemented directly: sessionTag packing, 5-REPL cap + spill, independent lifecycle and slot reuse. Statements use Sail by default, so engine compatibility is limited to the Spark Connect subset | 🟢 Real (default engine) |
| Environments | Applied to the session, not just reported. A session carries an environmentId; the item’s requirements.txt, Spark properties and JAR declarations are resolved and handed to the agent, which installs the packages and applies sparkConfig before the first statement runs. Witnessed by e2e/environment (CI job environment), which imports a package the runtime image does not have — and asserts the same import fails on a session with no Environment, so a package that happened to be in the image cannot make it pass. JARs are reported as skipped: a Spark Connect session’s classpath is fixed at engine start. One Environment per agent process — a second bind of the same one is a no-op and a different one is refused with a reason, because Fabric isolates per container and a single process cannot; letting the last bind win would corrupt a dependency tree. /opt/wheels remains the fallback for consumers who model no Environment. 37-runtime-fidelity-gaps.md | 🟢 Real (packages + Spark config; JARs out of scope) |
| Spark Job Definitions | The emulator runs the job, as Fabric’s pool does: V1 definition/main/arguments/libraries parsing, attached lakehouse + Environment resolution, then execution on the configured Spark agent with sys.argv = main file + the definition’s arguments, and the real outcome finalising the job and publishing its terminal event. Previously the emulator resolved the job and waited — only a client that fetched the source and ran it itself ever executed one. A JAR-bearing Environment is refused before any user code runs on an engine with no JVM (asked of the engine, not assumed), and accepted on the JVM overlay. With no agent configured the original callback contract stands unchanged: the job stays open for an external engine to report. Both halves are witnessed where each is real: e2e/livy runs with a Spark agent, so the emulator IS the pool there — it publishes a definition, submits the job and polls, and sjd-result=5 proves the engine read the bound lakehouse (3 rows) and received the definition’s arguments (2). e2e/notebook-run deliberately runs with no agent, so it remains the witness for the callback contract, with its runner playing the external engine | 🟢 Real (emulator-executed) |
Notebook code on the default engine (LakeSail’s Sail)
Section titled “Notebook code on the default engine (LakeSail’s Sail)”The engine behind the agent is Sail (Rust
Spark-Connect, no JVM). Every row is probed in CI (e2e/sail), not
inferred — the fidelity deltas a Fabric notebook author actually hits.
A Spark 3.5 JVM image (docker/spark-runtime, Fabric Runtime 1.3’s engine
baseline) exists as a CI compatibility oracle (e2e/spark-jvm), and the
statement agent still has its classic-session path. It is also exposed as a
user-facing overlay — docker compose -f docker-compose.yml -f docker-compose.override.yml -f docker-compose.spark-jvm.yml up swaps the
statement agent onto it — so the JVM-only rows below are graded 🟠
(🟠 non-default engine): real with that overlay attached, unavailable on the default
engine. Verified live, not inferred: sc.parallelize([1,2,3,4]).map(x*2) .sum() returns 20 through the Livy agent, and spark._jvm.io.delta.tables .DeltaTable resolves.
| Notebook pattern | Emulator (Sail) | Type |
|---|---|---|
abfss://…@onelake.dfs.fabric.microsoft.com/… production paths | Work unmodified (endpoint override routes the Hadoop URL form) | 🟢 Real |
| Delta write/read/append; SQL over temp views | Full | 🟢 Real |
Time travel — DataFrame option and SQL VERSION AS OF | Both forms work on Sail. The SQL form was previously graded 🔴 on the strength of an assertion; the generated engine matrix probes it against both engines and it passes on each | 🟢 Real |
MERGE INTO | Works against a registered table target (CREATE TABLE … USING delta LOCATION); path-based delta.`az://…` merge targets don’t resolve | 🟢 Real (registered) / 🔴 path target |
createDataFrame(local_rows) | Works (runners preset localRelationSizeLimit) | 🟢 Real |
sc / RDD API / spark._jvm | Fidelity inversion: works on real Fabric, impossible on Spark Connect — the agent binds sc to a guide-rail stub that raises a clear pointer instead of NameError. Restored by the JVM overlay (classic session): sc.parallelize(…).map(…).sum() verified | 🔴 default / 🟠 JVM overlay |
DML row-count envelopes (INSERT/MERGE counts) | The engine’s row count reaches the client: DataFusion reports it as uint64, which the Arrow conversion to the Connect client rejects, and the agent used to absorb that failure as an empty envelope — “wrote 3 rows” indistinguishable from “wrote nothing”. It now recovers the count from the statement’s cached result relation, which does not re-run the statement (measured: a 3-row INSERT stays 3 rows through recovery, and the e2e asserts the count-back for exactly that reason) | 🟢 Real (recovered count) |
| Structured streaming | Partially supported on Sail v0.6.6, and the detail matters: readStream plans (its schema resolves) and the streaming query manager works, but neither is evidence that data flows: a readStream is unobservable without a sink, and the console sink writes to the server’s stdout where no client can see it — driving it for six seconds produced no output at all. Sail also reports no progress metrics (lastProgress is None), so there is nothing else to interrogate. No durable sink works, and those are the ones whose output can actually be verified. memory → No table format found for: memory; parquet/csv → cannot write streaming data to listing table; delta → unsupported extension node for streaming: DeltaWriteNode. Since a streaming job that cannot land data is not useful, this stays 🔴 for real work. The engine matrix’s streaming rows now assert that rows reach the sink — files read back, or a queryable table — rather than that a query object reported itself active, which is what previously made console look green. Full support on the JVM overlay | 🔴 durable sink / 🟠 JVM overlay |
OPTIMIZE / VACUUM (Delta maintenance) | Partly real on the default engine, via delta-rs: Sail’s planner has no such commands, so the Livy agent recognises those statements and runs them through delta-rs against the same table — verified compacting 3 files into 1 and vacuuming the superseded 3, rows intact. ZORDER/WHERE are refused, not ignored (they change what is compacted); use the JVM overlay for the full syntax. Proven against a real abfss://…onelake… table end-to-end through Livy REST (e2e/livy), with a negative control: the same statement with credentials forced off is refused, so the pass is not vacuous. delta-rs authenticates on its own account — a Connect client cannot read back the session’s bearer — minting from the same issuer with the same Storage audience, resolved per statement so a refreshed token needs no restart. One caveat remains, unchanged and by design: the executor differs from real Fabric (the emulator performs it, not Spark). See 20-lakesail-engine.md | 🟢 Real (delta-rs) |
Java/Scala UDFs, spark.jars | Out of scope for Sail by design, not a technical impossibility. Sail already embeds a foreign runtime — it links CPython via pyo3 (sail-python-udf) for Python UDFs — so a JVM could be embedded too. The real obstacle is that a Spark Java/Scala UDF is compiled against Spark’s own classes (InternalRow, Catalyst encoders), so running one needs Spark’s jars loaded — which is JVM Spark, exactly what Sail exists to avoid. Bytecode translation (GraalVM, TeaVM) does not help: the problem is Spark’s API surface, not executing bytecode. The JVM overlay is the answer | 🔴 default / 🟠 JVM overlay |
| Change Data Feed | Read is real on the default engine: spark.delta_change_feed(uri) serves a genuine change feed from OneLake through delta-rs (verified in e2e/livy, 2 versions → 2 rows). Writing a CDF-enabled table is not possible on Sail — its writer answers Unsupported table features required: [ChangeDataFeed] even with the property and the feature both named, so the table has to come from delta-rs or the JVM overlay. Exposed as a named helper rather than an interception of spark.read, so it is always obvious which engine answered | 🟢 read / 🔴 Sail-authored table / 🟠 JVM overlay |
spark.jars | Accepted but inert on Sail: JARs have no classloader. The JVM overlay has a real one (verified) | 🔴 default / 🟠 JVM overlay |
| Concurrent Delta overwrite writers | Two independent Connect sessions race at one barrier; one commits and the other receives a transaction failure from the conditional Delta-log create | 🟢 probed conflict rejection |
Data Warehouse (data-warehouse/)
Section titled “Data Warehouse (data-warehouse/)”| Fabric feature | Emulator | Type |
|---|---|---|
| SQL-analytics-endpoint semantics over lakehouse Delta | DuckDB runs real SQL (aggregation / join / filter), e2e | 🟢 Real (engine in e2e) |
| Warehouse item management | Full | 🟢 Real |
| T-SQL over TDS + Entra FedAuth | Pure-Go TDS front (internal/tds) terminates the FedAuth handshake (real Entra token, database.windows.net audience), then byte-splices the client’s post-login session to a real per-item SQL Server connection so the engine emits every token itself. Unmodified go-mssqldb and Microsoft ODBC Driver 18 (pyodbc) clients connect and run T-SQL — including RPCs, prepared statements, and transactions. Verified against a real SQL Server; Microsoft’s real dbt-fabric adapter passes debug/seed/run/test end-to-end (e2e/dbt-fabric/) | 🟢 Real (front + default sidecar) |
| Lakehouse SQL analytics endpoint — Delta → engine | The emulator reads the lakehouse’s Tables/<t> Delta in pure Go and reflects (CREATE+INSERT) it into the sidecar on connect, so SELECT hits real OneLake data (matches DuckDB), read-only (writes rejected). Not PolyBase — SQL Server reading Delta in place is a proven dead-end on the Linux container (a throwaway spike; see 16-warehouse-tds.md) | 🟢 Real (reflection) |
| Warehouse — read-write T-SQL | Client CREATE/INSERT/SELECT relay straight to the sidecar; the warehouse owns its data (no reflection) | 🟢 Real (relay) |
Fabric SQL Database (database/) — OLTP + OneLake mirror | Same read-write TDS/FedAuth path (its own SQL Server database), plus mirroring: POST …/sqlDatabases/{id}/refreshMirror snapshots every table to OneLake as Delta (real Parquet + _delta_log), so Spark / DuckDB / delta-rs query the operational data. Verified with a go-mssqldb-writes → mirror → Delta-reads-back e2e (gated). Continuous/CDC mirroring and write-back-to-Delta are the deferred edge | 🟢 Real (snapshot mirror) |
| Per-item isolation (each item = its own SQL Server database) | Lakehouse/Warehouse routed by type; per-item databases so they never collide | 🟢 Real |
| RBAC → SQL permissions | Workspace role enforced on connect: no role → rejected; Viewer → read-only; Contributor+ → read-write (warehouse) | 🟢 Real |
information_schema / sys.* introspection | Relays natively — reflected/warehouse tables are real SQL Server tables | 🟢 Real (relay) |
| Per-column type fidelity (real SQL types over the wire) | The splice forwards SQL Server’s own COLMETADATA, so every column carries its true native type over the wire (the re-encode fallback, used only by fake test backends, synthesizes INTN/FLTN/BITN and falls back to NVARCHAR text) | 🟢 Real (native) |
Nested CTEs (WITH x AS (WITH y AS …)) | Fabric supports standard, sequential and nested CTEs; the SQL Server sidecar rejects the nested form (Msg 156), which would make the emulator stricter than Fabric. The TDS layer closes the gap on the wire: it parses the WITH prefix with a real lexer (string literals, quoted identifiers, nestable block comments) and flattens the nesting into the sequential form the sidecar accepts, then re-encodes the batch. Statements Fabric itself refuses are rejected rather than rewritten — non-SELECT targets, DML or OPTION hints in a nested definition, same-level duplicate names, and references escaping their nesting scope (the last of these verified to otherwise succeed post-flattening, i.e. a real divergence, not a theoretical one). Cross-level name shadowing is refused by name rather than renamed, because rewriting references safely needs a parser this layer deliberately is not. Statements sent as parameters (sp_prepexec/sp_executesql) are rewritten too: the RPC parameter list is walked, the statement parameter re-encoded, and the result re-parsed and compared with the original before it is forwarded — every other parameter byte-identical, or the original goes instead. Unmodelled parameter types (TEXT/XML/UDT/SQL_VARIANT) forward untouched rather than being guessed at. Witnessed by examples/medallion-pyspark running dbt’s native accepted_values and relationships — which compile to nested CTEs — with no CTE-free substitute in the project, and by a parameterized-statement probe through the real ODBC driver (29-tsql-parity.md) | 🟢 Real (rewrite + refusals) |
CTAS (CREATE TABLE … AS SELECT) | Fabric/Synapse spell “materialise this query” as CTAS; SQL Server has no such statement and spells it SELECT … INTO, so the sidecar rejects valid Fabric T-SQL with Msg 156. The TDS layer rewrites it, splicing INTO before the first FROM at paren depth 0 — including inside the EXEC('…') dynamic SQL dbt-fabric actually ships, which a statement-level rewrite would miss. Fabric’s WITH (DISTRIBUTION = …) options are dropped (physical layout, not results). Witnessed by examples/medallion-pyspark building gold with dbt’s own +materialized: table (29-tsql-parity.md) | 🟢 Real (rewrite) |
Class B strict mode (-tsql-strict) | Refuses what real Fabric rejects but SQL Server accepts — recursive CTEs, triggers, synonyms, CREATE USER, SET TRANSACTION ISOLATION LEVEL/ROWCOUNT/IDENTITY_INSERT, FOR XML, IDENTITY(seed, increment), enforced key constraints, multi-column statistics, PREDICT, sp_showspaceused — so a locally green build means a Fabric-green build. Off by default: it removes capability, which is the operator’s call. Indexed views and FOR JSON-in-a-subquery stay unenforced, with the reason recorded | 🟢 Real (opt-in) |
| Connection by item name (vs GUID) | Workspace read from the server name (<workspace>.datawarehouse.fabric.microsoft.com), item resolved by display name; a GUID still resolves by id (back-compat). Verified with a real go-mssqldb client | 🟢 Real |
Data Factory (data-factory/)
Section titled “Data Factory (data-factory/)”| Fabric feature | Emulator | Type |
|---|---|---|
Data Pipeline control flow (If / ForEach / Until / Switch / Filter / Fail, expression language, dependsOn) | Pure-Go interpreter that really executes. Deactivated activities never run: state: Inactive records the reserved Inactive placeholder status, branching follows onInactiveMarkAs (a Failed mark steers UponFailure branches without failing the run), and no output fields are invented — previously both fields were dropped on parse and a deactivated activity executed, side effects included | 🟢 Real (orchestration) |
| Per-activity policy — retry + backoff + timeout | Applied to every activity type: policy.retry re-runs a failed activity (each retry from scratch; only the final outcome is recorded, carrying retryAttempt); policy.retryIntervalInSeconds is folded into the run’s durationInSeconds as virtual backoff; policy.timeout fails an attempt whose own virtual duration exceeds the limit. No real sleeping — backoff and timeouts are exercised in milliseconds on the controllable clock | 🟢 Real |
ForEach sequential / parallel (isSequential, batchCount) | Iterations run in array order (deterministic); the mode sets the reported wall-clock — sequential iterations add, a parallel batch costs its slowest — matching how real Fabric overlaps them | 🟢 Real |
List pagination (continuationToken) | On by default, as in Fabric: a list past the server page size returns an opaque continuationToken and an absolute continuationUri, with no client opt-in — so a client that ignores the token breaks here exactly as it would in production. ?maxPageSize may narrow a page, never widen it. The page size is a testing lever (-list-page-size / FABRIC_LIST_PAGE_SIZE): set it to 2 and every list forces the token loop on the spot, the same idea as the controllable clock for LROs. Witnessed by Microsoft’s fab CLI — a plain list (no page-size parameter) comes back paged, and paging through is asserted complete: no item twice, none missed, terminal page tokenless | 🟢 Real |
| Invoke pipeline (ExecutePipeline) | Resolves the referenced DataPipeline (GUID or name, optional other workspace) and runs it through a fresh interpreter — real recursive interpretation, one level deeper on the same engines. waitOnCompletion (default) gates the parent on the child’s terminal status; parameters flow into the child; a cycle or excessive nesting fails loudly | 🟢 Real |
| Pipeline → notebook activity (TridentNotebook) | Resolves the notebook reference and creates a real RunNotebook job instance the pipeline gates on — the pipeline→jobs linkage is real; the notebook’s cells execute on the Spark sidecar, and a job with cells outstanding has no clock-derived completion: it stays non-terminal until the engine reports, so a completed job means the cells ran. Cell execution is witnessed end-to-end by ci:medallion: the pipeline’s Notebook activity starts a genuine run, Sail executes it over Spark Connect, and the example asserts the engine-reported read/write set as the Notebook lineage edge — evidence that exists only if the cells ran, not inferable from the binding | 🟢 Real (chain + cells) |
queryactivityruns detail | Full | 🟢 Real |
| Activity-level lineage | Successful Copy execution persists its resolved workspace/item/path source→sink edge, returns it in activity output, and exposes workspace lineage for OpenMetadata ingestion | 🟢 Real (Copy) |
| Copy activity — OneLake → OneLake | Really moves the bytes through the storage layer: a file, or a directory subtree preserving structure; source/sink locations {workspaceId?, itemId, path} are expression-resolved (GUID or name); returns real filesWritten / dataWritten. External stores / format transformation are out of scope and fail loudly | 🟢 Real (in-family) / 🔴 external |
| Lookup activity — OneLake CSV/JSON/Parquet/Delta | Reads real rows from a CSV, JSON, or standalone Parquet file, or a lakehouse Delta table (Tables/<name>, auto-detected — no format hint needed) in OneLake; honors firstRowOnly; the result flows into @activity(…).output for downstream steps. Parquet/Delta reuse the warehouse’s own Parquet reader — a real Delta column keeps its native type (int/float/bool), not a stringified cell | 🟢 Real (CSV/JSON/Parquet/Delta) |
| GetMetadata activity — OneLake path | Stats a real OneLake path: exists / itemType / size / lastModified / childItems; a missing path honestly returns exists:false | 🟢 Real |
| Delete data activity | Really removes the data through the storage layer, so FileDeleted events fire and a Reflex trigger sees a pipeline Delete exactly as it sees an ADLS client’s delete; recursive removes the subtree, non-recursive removes direct-child files and leaves subdirectories; returns filesDeleted. A missing path fails loudly naming the path — held to the loud side while the real oracle is unmeasured, per the comment on deleteActivity | 🟢 Real (storage layer) |
| Script / SqlServerStoredProcedure / SqlPoolStoredProcedure activities | Run real T-SQL against a Warehouse/Fabric-SQL-Database item’s own SQL Server database — the same per-item backend the TDS endpoint and the SQLDatabase mirror share. Script runs each scripts[] entry (Query → real rows back, NonQuery → rows affected); SqlServerStoredProcedure calls a real stored procedure with named parameters. The target is named directly as {workspaceId?, itemId} (the emulator’s own scoped mapping — real Fabric’s linkedService/connection reference isn’t modeled), the same shape Copy/Lookup/GetMetadata already use — and sqlPool is accepted as a fourth spelling of that key, because SqlPoolStoredProcedure is the Synapse-dedicated-pool name for the identical activity (same storedProcedureName + storedProcedureParameters, differing only in how it names its target) and the Warehouse is the item that owns a real SQL Server database here. That type string used to fall to the dispatch default and report Succeeded without executing anything. Honest error without a warehouse SQL backend attached | 🟢 Real (scoped) |
HDInsight Spark activity (HDInsightSpark) | The submission protocol terminated locally, the code executed by the engine the emulator already runs — the Livy precedent (docs/20) applied to a second protocol: no HDInsight cluster is proxied to, the activity’s contract is answered here and Sail computes. The entry file is read from OneLake at rootPath/entryFilePath and executed, with arguments reaching it as sys.argv (argv[0] the entry file, as a submitted application sees it) and sparkConfig carried in. The witness asserts the entry file’s own code reached the engine, not that the activity returned Succeeded — mutation-checked: an activity that reports success without executing fails it. Refused by name, each with its cause: className (the agent executes Python statements and has no path that submits a Java/Scala main class, on either engine — a JAR library attaching on the overlay is a different capability), proxyUser (no impersonation model), and sparkJobLinkedService (an external store the emulator does not model). No engine attached → an honest failure naming the missing engine, never a claimed submission. The output carries executedBy so a run cannot be misread as a real cluster | 🟢 Real (protocol terminated, Sail computes) |
Azure Databricks activities (DatabricksNotebook, DatabricksSparkPython, DatabricksSparkJar) | Same bet as the HDInsight row: the submission contract is terminated locally and the engine the emulator already runs executes the code. A notebook task’s file is read from OneLake and run with its baseParameters bound as names (the way dbutils.widgets delivers them, which is what notebook code reads); a Python task’s file runs with parameters as sys.argv, argv[0] the file. The two deliveries are opposite and a test pins each against the other’s mechanism — mutation-checked. DatabricksSparkJar is refused by name on BOTH engines: it asks the emulator to execute a Java/Scala main class, and the Spark agent runs Python statements — there is no submission path for one. That is narrower than “no JVM”: a JAR library does attach on the overlay (the Spark Job Definition path probes for it), so the two boundaries are stated separately rather than as one, and the refusal does not point at a remedy that would not remedy. Refused too are libraries (installing on a cluster whose lifecycle the emulator does not own — an Environment item is the modelled way) and any dbfs://Workspace//Shared//Repos path, because reinterpreting a Databricks path as a lakehouse path would invent a mapping nobody wrote. Files address OneLake as <lakehouseItemId>/<path>. No engine attached → an honest failure; the output carries executedBy rather than a fabricated runPageUrl | 🟢 Real (notebook + python, protocol terminated) / 🔴 JAR task, both engines |
Azure Batch activity (ADF Custom) | Off by default, and that is the design. Every other compute activity here runs Python through the Spark agent — user code in the engine’s sandbox. A Custom activity’s command is a shell command: a process on whatever host runs the agent, which is a different kind of thing. The repo already has a position on it, stated for the terminal pane in internal/config/config.go: “a terminal is not another read; it is arbitrary execution”, and “Empty = the feature does not exist”. So without FABRIC_CUSTOM_ACTIVITY=shell the activity refuses by name and no command reaches the agent — asserted, not assumed: the witness fails if a single statement is sent with the gate closed, and the mutation that opens it prints the echo pwned that would have run. Enabled, the command executes in the agent’s container rather than the emulator’s process (blast radius is the engine, not the API), extendedProperties become environment variables as Batch documents, and the command’s own exit code decides the activity — a non-zero exit fails it with the code and the command’s stderr, and an unreadable report fails too, because an unknown exit status is not success. Refused even when enabled, each naming its cause: resourceLinkedService/folderPath (Batch stages resource files from a storage account; the emulator models no connections, and staging nothing would leave the command referencing files that are not there), autoUserSpecification (no user model — certifying a privilege behaviour that does not exist), referenceObjects (linked services and datasets serialised onto the node) | 🟠 Opt-in (FABRIC_CUSTOM_ACTIVITY=shell), refuses by default |
Azure ML activities (AzureMLExecutePipeline, AzureMLBatchExecution, AzureMLUpdateResource) | Refused by name — and the refusal is a correction, not a gap. These three type strings were absent from the dispatch switch, so they fell to its default and were reported Succeeded: a pipeline that scored a model was recorded as having scored it, and the next step read an output nobody wrote. They now fail loudly instead, each naming its own cause. Why these do not run when HDInsight, Databricks and Azure Batch do: every one of those names a thing to execute the emulator can get hold of — an entry file, a notebook, a python file, a command — so terminating the external submission protocol locally and letting the engine we already run compute is honest (the Livy precedent, docs/20). mlPipelineId is an opaque handle on a pipeline published in an Azure ML workspace; its steps are not in the definition, not in OneLake, not in any item held here, so there is nothing to terminate the protocol onto, and running some other artifact in its place would invent the published pipeline’s behaviour — the same objection that refuses a dbfs: path. The MLflow sidecar does not rescue it: AML’s run history is MLflow-shaped, so the emulator could write the run record, but a record marked finished for steps that never ran only relocates the fabrication into the tracking store, where it is harder to see. AzureMLBatchExecution and AzureMLUpdateResource additionally target ML Studio (classic), which Azure has retired. Skipping one deliberately is supported by a real Fabric feature rather than an emulator flag — state: "Inactive" with onInactiveMarkAs — and the witness runs that advice rather than only asserting the message contains it | 🔴 Refused by name (was a silent success) |
| Validation activity | Really waits for the data, which is the only version of this activity worth having: it polls a real OneLake path and passes only when the path is actually there, honouring minimumSize (a file must be at least that many bytes) and childItems (true = the folder holds at least one file, false = the folder is empty — kept tri-state, since absent and false are different assertions). Directories are implicit in OneLake, so a folder is judged by what is under it rather than by a row that does not exist; an activity that only looked one up would report a landing folder absent forever. The wait is measured on the virtual clock — no real sleeps — and sleep between attempts is clamped to the remaining time, so a 100-second sleep cannot overshoot a 3-second deadline. Failing names which predicate was still unsatisfied, not just that time ran out. Before this it fell to the dispatch default and reported Succeeded, which for a guard is the worst possible bug: the pipeline reads absent data with the guard’s blessing. The wait loop is unit-tested through a scripted clock that forces the subscribe-before-read interleaving, the same discipline the WebHook park had to learn | 🟢 Real (OneLake + virtual clock) |
Azure Data Explorer Command activity (AzureDataExplorerCommand) | Runs the control command on the real Kusto engine the emulator already hosts behind Eventhouse — no cluster is proxied to, and the command travels the same relay helpers the KQL data plane uses, so the two cannot drift on database isolation or first-use creation. The witness asserts the engine received that command against the isolated engine database for that KQL Database item, not that the activity returned Succeeded, and that the engine’s internal database name is mapped home to the Fabric display name before the output is handed back. A query is refused by name: the schema’s contract is control commands (they begin with a dot), and one relayed to /v1/rest/mgmt would fail with a message about the engine rather than about the mistake — nothing is relayed when it is refused. Naming a non-KQLDatabase item is refused too, because proceeding would create an engine database from the wrong item’s id and report success against it. commandTimeout bounds the call. Connection stand-in: ADF puts the cluster and database on a linked service; the emulator names the target as {workspaceId?, itemId} like the rest of the family. It too used to be reported Succeeded while nothing ran — a silently-successful .drop table is the version worth picturing | 🟢 Real (protocol terminated, Kusto computes) |
| HDInsight Hive/Pig/MapReduce/Streaming, U-SQL, SSIS package activities | Refused by name, each with its own cause — and, like the Azure ML row, this is a correction rather than a gap: all six fell to the dispatch default and were reported Succeeded. The default is right for a connector leaf (a ServiceNow source really was reached in dependsOn order with its inputs resolved) and wrong for a compute activity, whose whole point is an effect later steps consume. HiveQL is refused because the agent routes SQL to Spark SQL, a different dialect — the overlap between them is not the same as the guarantee; Pig because nothing here interprets Pig Latin; MapReduce because it asks the emulator to execute a named Java main class, the same boundary the Databricks JAR task is refused at and on both engines; Streaming because there is no Hadoop Streaming harness (and its arbitrary-process half is the posture FABRIC_CUSTOM_ACTIVITY gates); U-SQL because Azure has retired Data Lake Analytics; SSIS because the emulator hosts no integration runtime and a package’s work is defined inside the package. How the set was found is the reusable part: the 41 discriminators in ADF’s published schema were diffed against what the dispatch switch and the pipeline interpreter actually handle | 🔴 Refused by name (were silent successes) |
Functions activity (AzureFunctionActivity) | Makes the real call to functionAppUrl + /api/ + functionName over the same HTTP core as Web (shared httpActivity, so the two cannot drift on timeouts, bounds, the non-2xx rule or output shaping); the function key travels as x-functions-key, and the witness’s stand-in rejects a wrong key so the pass cannot be vacuous. The ADF schema’s own rules are refused by name: method outside its seven-value enum, body on GET (“not allowed”), no body on POST/PUT (“required”). Connection stand-in: in ADF the URL and key live on the AzureFunctionLinkedService; the emulator models no connections, so both inline in typeProperties, and the error for a missing functionAppUrl says exactly that | 🟢 Real (connection inlined) |
| Web and WebHook activities | Makes the real HTTP call: method, URL, headers and body all resolved as expressions; a JSON response is merged into the activity output at the top level, so @activity('X').output.field resolves downstream; ADFWebActivityResponseHeaders and statusCode carried as Fabric does; a non-2xx fails the activity, matching Fabric rather than handing the status back; policy.timeout becomes the request deadline. Bounded at 8 MiB — the output lives in the run record, so it is not a download mechanism. FABRIC_WEB_ACTIVITY=stub restores the old record-without-calling behaviour for a hermetic CI leg, and labels the output stubbed: true so such a run cannot be mistaken for one that called | 🟢 Real (both) — WebHook parks for real: the call carries a generated callBackUri (a PATH, per the repo’s the-path-is-the-contract precedent; the token in it is the credential, as ADF’s own), the pipeline goroutine parks until the receiver’s callback or a timeout measured on the virtual clock (a frozen clock advanced past the deadline expires it with no real sleep — clock.Changed exists for exactly this, and the witness fails without it), callback body fields surface in the activity output, and reportStatusOnCallBack consumes a reported non-2xx as the activity’s failure. ADF schema enforced by name: POST-only enum, object body, D.HH:MM:SS timeout |
REST connector — RestSource and RestSink in a Copy, with pagination | Makes the real HTTP requests and commits the rows: url (or the linked service’s url + dataset’s relativeUrl), requestMethod GET/POST, requestBody, additionalHeaders and httpRequestTimeout, all expression-resolved. The JSON response is shaped into a Delta table in the sink lakehouse — translator.collectionReference picks the record array, and a response holding exactly one array needs no hint; translator.mappings lift fields out of nested records by JSONPath — the shape BMC Helix actually returns (entries[].values.*), where auto-flatten finds no scalar columns at all. Witnessed end to end by e2e/rest-helix: a Web activity logs in for an AR-JWT, the copy pages limit/offset through a stand-in Helix that rejects Bearer, and the incidents land as Delta. paginationRules really page: AbsoluteUrl (absolute or relative) and QueryParameters.x / Headers.x cursors read out of the body (JSONPath) or the response headers; {placeholder} selectors stepped by RANGE:start:end:step (an empty end is open-ended) — the ServiceNow/BMC Helix limit+offset shape; EndCondition: with Empty / NonExist / Exist / Const:v; MaxRequestNumber; and RFC 5988 Link: …; rel="next", on by default only when no other rule is declared, as Fabric documents. The loop also stops on HTTP 204 and on any JSONPath resolving to null. A rule the emulator does not implement is refused by name rather than ignored. Accept is forced to application/json as Fabric does; a non-2xx fails the copy; a top-level JSON array with pagination rules is refused, since Fabric does not paginate that shape. Nested fields have no column shape and are reported by name in skippedColumns. Bounded at 8 MiB per page, 1M rows and 1000 pages — refused, never truncated, so a next that points at itself fails loudly instead of looping. Auth is Anonymous + additionalHeaders; the emulator models no connections, so any other authenticationType fails by name — a Web activity can fetch a token and pass it as an expression, which is how BMC Helix’s AR-JWT scheme works here. RestSink writes the other direction: rows from a Delta table or a Parquet/CSV file are POSTed (or PUT/PATCHed) in batches of writeBatchSize (default 10000), each request carrying Fabric’s payload — a JSON array of row objects. httpCompressionType: gzip, additionalHeaders, httpRequestTimeout and requestInterval (validated against Fabric’s [10, 60000]ms band, and virtual like retry backoff) are honoured; a non-2xx fails the copy, zero rows send no request rather than an empty array, and a BinarySource is refused because opaque bytes have no rows to send | 🟢 Real |
Salesforce — SalesforceV2Source and SalesforceV2Sink in a Copy | Runs the real Bulk API 2.0 query lifecycle: creates a query job (operation is queryAll when includeDeletedObjects is set — a different operation, not a filter), polls it to a terminal state, then downloads CSV result sets paged by the Sforce-Locator header, which ends on the literal string "null". objectApiName or a SOQL query; with neither, all the object’s data. A Failed/Aborted job fails the copy naming the job id so it is findable in the org’s own monitor. reportId is refused by name — a report is the Analytics REST API, not a Bulk query, and running a query instead would return the object’s rows rather than the report’s. The org is named directly (instanceUrl + accessToken, both expression-resolved) because the emulator models no connections; a Web activity can run the OAuth call and pass the token in. Bounded at 8 MiB per page, 1M rows and 1000 pages. Sink (SalesforceV2Sink) is scoped, not built — 41-salesforce-connector-plan.md. SalesforceV2Sink writes the other direction through the ingest lifecycle: create a job, PUT the CSV to its contentUrl, PATCH it to UploadComplete (a job left Open uploads rows nothing ever processes), then poll. One job per writeBatchSize (default 100,000); writeBehavior Insert or Upsert, with externalIdFieldName required for upsert and refused without it. ignoreNullValues picks between Bulk CSV’s two meanings — an empty field leaves the value unchanged, the literal #N/A sets it to NULL — and an empty string stays an empty string rather than becoming either. A job that reaches JobComplete with records rejected fails the copy, naming failedResults, because a partial write reported as a whole one is the worst outcome available. Witnessed end to end by e2e/salesforce: a round trip — OAuth via a Web activity, five accounts out over three locator pages into Delta, then upserted back in three ingest jobs, with every record compared against what the org originally served | 🟢 Real |
ServiceNow via RestSource | Real, and it is the case Microsoft documents. api/now/table/incident?sysparm_limit=…&sysparm_offset=… is Example 1 of the REST connector’s pagination rules, with "QueryParameters.{offset}": "RANGE:0:10000:1000" — the emulator runs that example. Both documented paging routes are witnessed in one pipeline (the offset RANGE and RFC 5988 Link headers), rows land as Delta read back through OneLake, and the negative controls refuse an anonymous and a wrong-credential read so a pass cannot mean the target let anyone in. Records are FLAT under $.result, so this carries no translator.mappings and auto-flattening must infer the columns — the half e2e/rest-helix cannot reach, since ARS nests under values. Scope: this is the Table API’s documented shape reached through RestSource, not Fabric’s first-party ServiceNow connector type, which the emulator does not implement; auth is Basic through additionalHeaders because a native authenticationType is refused by name (no connection model). That is the documented reason a real user reaches for RestSource here — the built-in connector is Basic-only, so OAuth needs this route | 🟢 Real (against a modelled Table API) |
| External-connector leaves (ServiceNow, …) | Stubbed success — reached in dependsOn order with inputs resolved, but nothing executes: each needs a vendor SDK and credentials the emulator has neither of. The REST connector above is the working route to any of them that expose a REST API — and to BMC Helix, which real Fabric has no connector for at all | 🟡 Emulated |
| Apache Airflow Job | Typed item + beta file APIs; uploaded Python DAGs sync to an opt-in real Airflow 2.10.5/Python 3.12 sidecar, whose scheduler/executor and REST state determine the Fabric job result (e2e/airflow) | 🟢 Real (sidecar) |
| Dataflow Gen2 (Power Query M engine) | Typed item/definition management round-trips. Refresh, Publish, and in-pipeline execution fail with DataflowEngineNotImplemented; no open Power Query M engine exists to attach | 🟡 mgmt / 🔴 exec |
| Connectors / on-prem gateways | — | 🔴 Not implemented |
CI/CD (cicd/)
Section titled “CI/CD (cicd/)”| Fabric feature | Emulator | Type |
|---|---|---|
| Git integration (connect / status / commit / update / disconnect) | Full, real state | 🟢 Real |
fabric-cicd tool publishing | The real client round-trips definitions (e2e) | 🟢 Real |
| Deployment pipelines — model, assignment, item pairing, Deploy Stage Content (D0–D2) | Real promotion: definitions really copy, pairs decide (not names), metadata only — a deployed lakehouse arrives empty — and target-only items survive. 202 LRO + /result detail. 23-deployment-pipelines.md | 🟢 Real |
| Deployment pipelines — role-assignment CRUD (D3) | Add / Delete / List; Admin is the only role a pipeline defines; mutations require Admin, reads require membership | 🟢 Real |
Item types present, engine absent
Section titled “Item types present, engine absent”| Fabric area | Emulator | Type |
|---|---|---|
Real-Time Intelligence — Eventhouse / KQL Database (real-time-intelligence/) | Full item management (including the default child database an eventhouse creates, and creationPayload.parentEventhouseItemId), plus the Kusto REST protocol on the eventhouse’s published properties.queryServiceUri — /v1/rest/mgmt, /v1/rest/query, /v2/rest/query — terminated by the emulator (Kusto-audience bearer, workspace RBAC, one isolated engine database per Fabric KQL Database) and executed by Microsoft’s own KQL engine container (kustainer) when the rti profile attaches it. No engine attached → honest 501. 25-rti-kusto.md | 🟡 mgmt / 🟠 exec (opt-in --profile rti) |
Real-Time Intelligence — Eventstream (real-time-intelligence/event-streams/) | Item management only. The attached Kusto engine is a query/ingest engine with no streaming ingestion — a streaming pipeline is a different service, deferred with cause | 🟡 mgmt / 🔴 exec |
CopyJob | A Copy Job copies. jobType=Execute (Microsoft’s documented run-on-demand spelling; the readback says CopyJob, and both dispatch) parses copyjob-content.json and runs each activity through the same pipeline Copy executor — real bytes into the sink, real Delta commits for Tables/ sinks (Append appends, Overwrite replaces), and a lineage edge with producer Copy. The copy’s completion decides the status, not the clock — run inline, a finished copy reported InProgress for the whole LRO-delay window, which is what the witness pins (1h delay vs a 5s poll deadline, destination bytes asserted present). Dispatch is async like every other executing type, so the POST does not hold a socket for the copy’s duration; that half is true by construction and not separately witnessed — a blockable copy would be needed, and OneLake-to-OneLake has no seam to block on. The boundary is refused by name, not skipped: external connections (CopyJobExternalSourceNotSupported / …DestinationNotSupported, source checked first so a both-external refusal is deterministic), jobMode: CDC (CopyJobCDCNotImplemented), Merge/Upsert (CopyJobWriteBehaviorNotSupported) | 🟢 Real (Lakehouse legs) / 🔴 external, CDC |
KQLDashboard, KQLQueryset, WarehouseSnapshot | Typed collections over the generic item surface — create/get/list/patch/delete and definition round-trip, with the collection forcing its type. Type names taken from fabric-docs payloads; no execution engine for these | 🟡 mgmt / 🔴 exec |
Event triggers — Reflex (Data Activator) on OneLake events | The Reflex triggers. A trigger subscribes to a source item’s OneLake file events (FileCreated / FileDeleted / FileRenamed) under an optional path prefix, and a match starts a real item job — invokeType: "EventTriggered" — with the event bound in as @pipeline()?.TriggerEvent?.FileName / .FolderPath / .Subject, reading which needed safe navigation (?.) in the expression language, so one definition works whether or not a trigger started it. No broker is needed: every byte written to OneLake passes through the emulator’s own storage layer, so a file event is observable at the source whoever wrote it — an ADLS client, azcopy, delta-rs, a Copy activity, the mirror writer. Dispatch is synchronous and reentrant, so bronze→silver→gold chains work; a runaway is bounded by an activation count rather than by identity, matching how Fabric bounds one — Activator documents Fabric item — Activations/user/minute — 50 and throttles or cancels beyond it, and caps input at 10,000 events/second/rule, with no documented loop detection or dedup. Independent events each activate, because Activator “continues monitoring without waiting for the action to complete”. Boundary: the binding is an emulator-native control surface (…/reflexes/{id}/triggers) because Fabric has no public REST for it — the Eventstream/Reflex rule is assembled in the portal. What is faithful is everything downstream: the filter, the invocation, the TriggerEvent fields, and a real pipeline really running. 07-control-plane-api.md | 🟢 Real (OneLake sources) |
Mirroring — Mirrored Database (mirroring/) | POST …/mirroredDatabases/{id}/refreshMirror mirrors an external SQL Server source (reached via a Connection with Basic credentials) to OneLake as real Delta — reusing the exact same mirror writer the Fabric SQL Database uses (warehouse.Mirror; same code, external source). Proven by a gated e2e: a table seeded directly on an external database (bypassing the emulator’s own per-item routing entirely) mirrors and reads back correctly. Snapshot-on-trigger, not continuous/CDC replication; other source engines (Snowflake, CosmosDB, on-prem via gateway) are out of scope | 🟢 Real (snapshot mirror, SQL Server sources) |
Power BI — Semantic Model query (executeQueries) | Real bounded DAX engine — EVALUATE, SUMMARIZECOLUMNS, measures, SUM/DIVIDE/COUNTROWS/IF/SELECTEDVALUE, the infix operators (+ - * / &, comparisons) with DAX precedence, relationship filter propagation, and a name that does not resolve erroring rather than aggregating to zero — over imported data.json or compatibility-level-1604 Direct Lake entity partitions backed by current OneLake Delta. Proven by the golden DAX/GX suites and the Spark-written Direct Lake witness. | 🟢 Real (DAX subset + Direct Lake) |
| Power BI — Report items | Typed item + definition round-trip. Rendering is deliberately absent and is not a gap: a Power BI report is drawn client-side from a definition plus query results, and the emulator serves both halves — the definition here, the DAX through executeQueries. There is no server-side renderer in Fabric for this to be unfaithful to | 🟢 Real (definition round-trip; rendering is client-side) |
| Power BI — DAX beyond the bounded subset | internal/semanticmodel answers the subset its golden fixtures pin (EVALUATE, SUMMARIZECOLUMNS, measures, SUM/DIVIDE/COUNTROWS/IF/SELECTEDVALUE, infix operators with precedence, single-hop relationship propagation) and errors rather than mis-evaluates outside it. “Full DAX” is not a reachable state; what is reachable is a measured coverage figure against a real engine, and the oracle now exists — e2e/pbix-desktop runs Power BI Desktop itself on windows-latest and agreed with executeQueries bit-identically (rel = 0.000e+00, 5/5 runs). Growing the evaluator is now evidence-driven rather than docs-driven; see 33-pbix-tooling.md | 🟡 bounded subset, oracle available |
| Power BI — SemPy / semantic-link-labs over XMLA | Deferred on cost, not feasibility — and that distinction was measured, not assumed. e2e/xmla drives Microsoft’s own ADOMD.NET on Linux/.NET 8 and established off the wire that the endpoint IS overridable (powerbi://host:port), that a self-signed CA is trusted, and that a bearer from the connection string works — three claims that had blocked this as impossible. The engine is not the missing piece: executeQueries and XMLA are two envelopes around the same evaluator, so what XMLA needs is a transport. Phased in 32-xmla-plan.md | 🔴 not implemented (feasibility measured, cost unpriced) |
Data Science — ML models / experiments / MLflow (data-science/) | Authenticated, workspace-scoped proxy to a real MLflow 3 tracking/model-registry server. Experiment/model creation synchronizes typed Fabric items; experiment/run references are isolated by workspace; successful artifact uploads are mirrored under the experiment item’s OneLake Files/mlflow-artifacts. | 🟢 Real (sidecar) |
Graph (graph/), Real-Time Hub, Copilot / IQ (iq/), Embed, Workload Dev Kit | — | 🔴 Not implemented |
Emulator-only (no Fabric equivalent — these exist for testing)
Section titled “Emulator-only (no Fabric equivalent — these exist for testing)”| Capability | Purpose |
|---|---|
Controllable clock (/_emulator/clock) | Advance virtual time to drive LRO / job status transitions deterministically. |
Fault injection (/_emulator/faults, /_emulator/permissions) | Force failures / throttling / RBAC denials to test client resilience. |
Flow stream (/_emulator/events) | Server-Sent Events for every byte that moves through OneLake — file events for writes/renames/deletes whoever made them, and table events derived from Delta commits (version, rowsAdded, filesAdded). Plus job and activity events — a failing activity reaches the stream with its error and job id the moment it fails, rather than being reconstructed from queryactivityruns afterwards — and attribution: a Copy activity’s writes name the activity, a notebook cell’s name the cell (including engines that cannot set headers, via the same bearer claims lineage uses). Never inferred; a write with nothing to say carries no attribution. curl -N tails a medallion run live; ?kinds= narrows it and ?since= replays the ring buffer. Delivery is deliberately lossy — a slow consumer is told what it dropped rather than being allowed to stall a writer. Two further kinds complete a medallion’s chain: lineage when a movement is recorded (so a graph redraws as a warehouse build happens, which has no job to end), and query when a semantic model is read — the Power BI hop, which is an event and never an edge because a query moves no data. 31-flow-observability.md A job’s terminal event is checked STRUCTURALLY, not type by type. startJob used to decide the outcome in one place and announce it from a second list, and a disagreement between the two was silent: the status converged (the clock, or a FinalizeJob call) while nothing reached the stream, so every status-polling test passed and the Flow view showed a job running forever. Writing the structural test found a live instance — an Apache Airflow job that could not start (no dagId, or no Airflow attached) was recorded Failed and announced to nobody, because the “failed to even start” branch listed Notebook and Spark job definition and not it. The second list is now gone: startJob records what it settled where it settles it, and the deferred announcement reads that, so the decision and the announcement cannot drift. The test walks every dispatched (item type, job type) pair and pins three states rather than one — announced now, left to the clock (a generic item, deliberately unannounced), and awaiting an engine (not terminal and must not be announced as though it were, which is the opposite failure and just as silent). |
| Data-flow lineage beyond OneLake | A medallion’s last hops move no bytes through OneLake, so each is observed on its own terms rather than inferred. silver → gold: the TDS front already parses every statement for dialect adaptation, so it also records what the engine accepted — CTAS, SELECT … INTO, INSERT … SELECT, CREATE VIEW, sp_rename, DROP, including inside the EXEC('…') dbt-fabric actually ships — resolving three-part names to Fabric items (producer: Warehouse). dbt’s build-then-swap is followed through sp_rename, so the graph names fct_order_lines and not its __dbt_temp scaffold. A statement whose response carries an error, or whose outcome cannot be read, records nothing. gold → semantic model: a Direct Lake table’s binding names its source, so the edge is recorded when the definition lands (producer: DirectLake); an import model gets none, because its rows arrive detached from wherever they were selected and inventing a source would be a guess. Interactive engines: POST /v1/workspaces/{wid}/lineage is the notebook read/write report without the job, for a Spark session or plain script (producer: Reported) — emulator-native, as Fabric has no such endpoint. The producer is the point: Warehouse is evidence, Reported is a claim, and a consumer never has to guess which. 31-flow-observability.md |
| Svelte management portal | Dashboard, workspaces, operations, clock, and fault controls. |
Ecosystem conformance: real OSS/vendor clients as witnesses
Section titled “Ecosystem conformance: real OSS/vendor clients as witnesses”Parity isn’t claimed from our own tests alone — each 🟢 surface is pinned against
the real, unmodified client a Fabric user runs, executed against the emulator
in CI (e2e/<client>/). If Microsoft’s own tool round-trips unchanged, the
contract holds better than any assertion we could write ourselves.
| Real client (pinned) | Surface exercised | Status |
|---|---|---|
fabric-cicd (Microsoft) | Control plane / CI-CD publish | 🟢 e2e/fabric-cicd |
Fabric CLI fab (Microsoft) | Control plane — SPN auth (MSAL) + workspace/item CRUD (Notebook, SemanticModel, Report, DataPipeline, Lakehouse), ls/get/api | 🟢 e2e/fabric-cli |
| Fabric Data Engineering VS Code extension 1.18.1 contract (Microsoft) | Shared-backend/MWC authoring routes through api.powerbi.com; interactive kernel websocket is not claimed | 🟢 e2e/vscode-extension |
| Apache Airflow 2.10.5 | ApacheAirflowJob DAG discovery, scheduling, execution, and status | 🟢 e2e/airflow |
| MLflow 3 + dbt-duckdb | Workspace-scoped experiment/run/artifact/model lifecycle, followed by dbt’s real Delta plugin over the same Spark-written OneLake table | 🟢 e2e/data-science-loop |
deltalake (delta-rs) | OneLake Delta write/read | 🟢 e2e/delta-rs |
azure-storage-file-datalake + Blob SDK | OneLake ADLS Gen2 DFS + Blob | 🟢 e2e/adls-sdk |
azcopy (Microsoft) | OneLake Blob multi-block transfer | 🟢 e2e/azcopy |
| DuckDB | Lakehouse SQL over Delta/Parquet | 🟢 e2e/duckdb |
| PySpark behind the Livy API | Spark sessions / statements | 🟢 e2e/spark, e2e/livy, e2e/notebook-run |
notebookutils | Notebook utility shim | 🟢 e2e/notebookutils |
go-mssqldb | Warehouse/Lakehouse TDS + FedAuth | 🟢 internal/server, internal/tds |
dbt-fabricspark (Microsoft) | Fabric Spark via Livy HC sessions | 🟢 e2e/dbt-fabricspark — debug→seed→run→test on Sail |
dbt-fabric (Microsoft) | Warehouse TDS via ODBC Driver 18 | 🟢 e2e/dbt-fabric — debug→seed→run→test through the TDS splice |
azure-kusto-data (Microsoft) + raw Kusto REST, over kustainer (Microsoft’s own KQL engine) | Eventhouse / KQL Database: /v1/rest/mgmt, /v1/rest/query, /v2/rest/query on the published queryServiceUri — create table, ingest, query values back, per-database isolation | 🟠 e2e/rti — witness of record is CI (amd64). The engine needs AVX2, which Rosetta does not provide, so the default Docker setup on Apple silicon cannot run it; a QEMU x86-64 VM with --cpu-type max can, and does (25-rti-kusto.md) |
The TDS surface now has two independent driver witnesses: go-mssqldb and
the Microsoft ODBC Driver 18 (via dbt-fabric). That second driver mattered —
it exposed a real gap: go-mssqldb tolerated a synthesized FedAuth login, but
ODBC Driver 18 took a compatibility path (prepared-statement RPCs +
sp_reset_connection under mandatory connection pooling) that desynced against a
re-encoding relay. The fix was to byte-splice the post-login session straight
to the real SQL Server (so it emits every token itself), which is exactly the
kind of driver-family gap a single-driver test never surfaces. dbt-fabricspark
likewise drives the high-concurrency Livy layer over its real Livy-session
protocol (method: livy, service-principal auth via entra-emulator).
Scope boundary: Fabric, not the predecessor Azure products
Section titled “Scope boundary: Fabric, not the predecessor Azure products”The emulator targets Microsoft Fabric — the convergence/successor product — not the earlier Azure analytics services Fabric replaced. That boundary is why some adjacent dbt adapters and Azure surfaces are intentionally not built: they belong to predecessor (often retired) products, and their Fabric-native successors are what we emulate instead.
| Adjacent product / client | Why out of scope | Fabric-era equivalent (in scope) |
|---|---|---|
Azure Synapse dedicated SQL pool (dbt-synapse) | Different product: its own control plane (Synapse workspaces) and an MPP T-SQL dialect (DISTRIBUTION = HASH, clustered-columnstore / resource-class DDL) that our vanilla SQL Server sidecar rejects. dbt-synapse layers on dbt-fabric, so the shared SQL path is already covered by the dbt-fabric witness | Fabric Warehouse — 🟢 TDS relay |
Azure Data Lake Analytics — U-SQL / SCOPE (dbt-scope) | Retired service (EOL Feb 2024), proprietary batch language, no Fabric embodiment. The only overlap (Delta on a lake) is Spark/OneLake, already witnessed | Fabric Spark — 🟠 Livy |
| ADLS Gen1 | Retired (Feb 2024), superseded by Gen2 | — |
| ADLS Gen2 (standalone storage account) | Not missing — OneLake is the Gen2 endpoint: hierarchical namespace, the dfs filesystem API, onelake.dfs.fabric.microsoft.com. Fabric has no separate storage account to emulate | OneLake — 🟢 e2e/adls-sdk |
Rule of thumb: if a capability exists only in a product Fabric replaced, it’s out
of scope; its Fabric-native successor is what we build. “We already have the
TDS/SQL Server foundation” makes Synapse cheaper, not done — the remaining
delta is a whole MPP dialect plus a second control plane, for a superseded
target. So the two dbt adapters we build (dbt-fabricspark, dbt-fabric) are
exactly the two that hit live Fabric surfaces; the other two (dbt-synapse,
dbt-scope) target predecessor products outside the emulator’s remit.
Why the boundary sits where it does
Section titled “Why the boundary sits where it does”Real Fabric’s own Livy endpoint is Microsoft’s implementation of the Livy REST contract over their Spark platform — they honor the protocol, not the retired Apache Livy server. And where Fabric adds its own layer on top of that protocol — high-concurrency REPL packing, which a vanilla Livy server has no concept of — the emulator implements that layer directly rather than proxying, because there is nothing to proxy it to. That is the same stance throughout: the protocol and control plane are the durable, real things (built, not mocked, so real clients can’t tell the difference), and the compute engine is attached (Spark; T-SQL on SQL Server; KQL on Microsoft’s own Kusto engine — 25-rti-kusto.md) or deferred when proprietary or without an implementation to attach at all (Dataflow Gen2’s M engine, Power BI rendering, Eventstream’s streaming ingestion). Every deferral fails loudly rather than pretending to succeed. See 13-roadmap.md for the milestone history and the deferred-with-cause rationale.