Roadmap
Scope chosen: full — control plane through the OneLake data plane, matching entra-emulator’s surfaces (portal + Starlight docs + distribution), composed via docker-compose. Each phase is independently useful and CI-verified (real-SDK e2e, like entra-emulator’s SDK matrix).
P0 — the spine (token acceptance + workspaces + items + RBAC + LRO)
Section titled “P0 — the spine (token acceptance + workspaces + items + RBAC + LRO)”The minimum that lets someone test SP → Fabric client-credentials automation.
- Token acceptance: validate Bearer against entra-emulator JWKS/issuer;
audience set;
oid/appidextraction. (--entra-issuer,--entra-jwks-url) - Store + migrations (
workspace,item,role_assignment,operation). - LRO engine on the controllable clock (
202/x-ms-operation-id/Location/Retry-After,GET /operations/{id}+/result). - Workspaces CRUD. (
assignToCapacitydeferred at P0; the capacity model is now designed in 07-control-plane-api.md (## Capacities) — seeded default capacity, auto-assign on create, assign/unassign LROs — validated as needed by fabric-cicd’s capacityId check. ARM consume is opt-inFABRIC_ARM_URL: ARM-createdMicrosoft.Fabric/capacitiesappear onGET /v1/capacities; standalone seed is unchanged.) - Generic items CRUD (create-with-definition → 202 LRO).
- RBAC: role assignments CRUD + enforcement (Admin/Member/Contributor/Viewer; creator becomes Admin; Member grants ≤ Member).
- Fault injection + clock control (
/_emulator/clock,/_emulator/faults). - Health; Docker image; docker-compose with entra-emulator.
- e2e: in-process entra-emulator mints a real client-credentials token for the Fabric audience; full workspace/RBAC/item/LRO flow over HTTP.
P1 — CI/CD (the primary draw)
Section titled “P1 — CI/CD (the primary draw)”Makes fabric-cicd, git integration, and deployment pipelines run offline.
- Item definitions:
getDefinition(200) /updateDefinition(202 LRO), parts round-trip verbatim. - Typed item aliases (12 collections: notebooks, lakehouses, warehouses, dataPipelines, semanticModels, reports, environments, eventhouses, kqlDatabases, sparkJobDefinitions, mirroredDatabases, eventstreams).
- Connections:
GET/POST /v1/connections— gitconnectwith a service principal requires aconnectionId(SPs may not useAutomatic). - Git integration: connect / initializeConnection / status / commitToGit / updateFromGit / disconnect / myGitCredentials, backed by a local per-branch definition store (logical ids preserved across commits; updateFromGit mirrors: creates, replaces definitions, deletes stale).
- Jobs:
jobs/instances?jobType=trigger (202 + Location) + clock-derivedNotStarted→InProgress→Completed/Failed+ cancel (Cancelled). - e2e: two-workspace git round-trip over HTTP (commit from one, update into another, definitions intact); job lifecycle on the frozen clock.
- e2e: the real
fabric-cicdPython tool (v1.3.x) publishes into the emulator —e2e/fabric-cicd/run.py(self-contained: both emulators + venv + driver). Works unmodified via its ownFABRIC_API_ROOT_URL/DEFAULT_API_ROOT_URLoverrides + in-process DNS pin (our TLS cert coversapi.fabric.microsoft.com). Driving it surfaced and fixed real gaps:/v1/workspaces/{id}/folders(now implemented),descriptionalways present on item wire shapes, result-less LROs must not advertise a result Location, and fabric-cicd refuses workspaces with nocapacityId. Remaining: wire into CI once the GitHub remote exists. - e2e: Microsoft’s Fabric CLI (
fab) drives the control plane —e2e/fabric-cli(containerized,fabv1.6+): service-principal auth (its MSAL flow against entra-emulator, which islogin.microsoftonline .comvia a compose alias +FAB_API_ENDPOINT_FABRICfor the API), then workspace + item CRUD (Notebook / SemanticModel / Report / DataPipeline / Lakehouse),ls/get, and the rawapipassthrough — all unmodified. Driving it surfaced and fixed a real gap in entra-emulator: MSAL/ADAL validate the authority viaGET /common/discovery/instancebefore every token, which returned 404; entra-emulator now serves it (v0.2.2). That same fix removes the root cause ofazcopy’s static-token workaround (its MSAL authority validation now succeeds against the emulator). - Deployment pipelines — the third item in this phase’s own header,
now shipped end to end (D0–D3).
D0 (pipeline/stage model + read surface: 2–10 ordered stages,
default Development/Test/Production, per-pipeline RBAC where the creator
is Admin and non-members get 404, stage→workspace assignment with live
name resolution and
ON DELETE SET NULL). D1 (assign/ unassign workspace + real item pairing: pairs are item-id edges between adjacent stages that survive renames on either side, recomputed only at assign — never lazily at read time). D2 (Deploy Stage Content over the existing LRO engine, deploy-all and selective, both directions between adjacent stages, per-item detail on/operations/{id}/result, plus the deployment-operations history). D3 (role-assignment CRUD; Admin is the only role a pipeline defines, mutations require Admin while reads require membership). e2e: Microsoft’sfabCLI drives the whole promotion flow (e2e/fabric-cli), following the call order of Microsoft’s ownDeploymentPipelines-DeployAll.ps1— list → stages → deploy → poll → result — which independently confirmed the wire contract. Fabric has exactly two CI/CD mechanisms, git integration is one and stage-to-stage promotion is the other. Designed in 23-deployment-pipelines.md (D0 model+read → D1 assignment+pairing → D2 deploy over the existing LRO engine → D3 role assignments). Most of the work is pairing, which is persistent state surviving renames — not a name match — and deployment copying metadata only, never data, and not deleting target-only items the wayupdateFromGitdoes. D1 turned the first open fidelity question into a measurement: the documented ambiguous-pairing case is structurally unreachable here — the28e4a4cUNIQUE index forbids duplicate name+type in a workspace, and items carry no folder membership, so the documented tie-breaker has no data.PairItemsis a pure function so that branch is still unit-tested against inputs the store cannot produce. The second question (whether deploy overwrites the target’s display name) stays open for the conformance oracle.
P2 — the identity handshake (deepest entra integration)
Section titled “P2 — the identity handshake (deepest entra integration)”The “works seamlessly with entra-emulator” payoff. Its dependency —
entra-emulator roadmap #16 — has already shipped: the workspace-identity
object (internal/store/fabric.go, states Active/Provisioning/Failed/ Deprovisioning, name-follows-workspace, cascade delete), admin CRUD at
/admin/api/workspace-identities, internal token minting at
GET /fabric/workspaceidentities/{id}/token, and acceptance of both Fabric
audiences. P2 can start any time; it consumes those endpoints over HTTP.
- Workspace-identity lifecycle:
POST /v1/workspaces/{id}/provisionIdentity/deprovisionIdentity(202 LRO) drive entra’s admin API over HTTP (internal/entraclient, origin derived from the issuer). Rename follows the workspace; workspace delete cascades the identity; the identity appears asworkspaceIdentity{applicationId,servicePrincipalId}on the workspace shape. - The provisioned identity’s SP is granted Admin on its workspace, so
tokens entra mints for it (
GET /fabric/workspaceidentities/{id}/token— customer never holds a credential) pass RBAC back here. Deprovision revokes the grant. - Audit event parity: entra-side — its token mint emits
Retrieved Fabric Identity Token for Workspace(covered by its tests). - e2e: provision → entra mints for the identity → the identity’s token reads its workspace and creates items in fabric-emulator; rename-follows verified in entra; deprovision revokes; workspace delete cascades.
P3 — OneLake data plane
Section titled “P3 — OneLake data plane”-
onelake.host mux (Host-routed like real Fabric): ADLS-Gen2 subset — PUT create file/directory, PATCH append/flush (position-checked), GET read, HEAD properties, filesystem listing (?resource=filesystem,directory=,recursive=, non-recursive collapses to first-level dirs), DELETE (directories take their subtree). -
Storage-audience token acceptance (separate validator over the same JWKS; fabric-audience tokens are rejected on the data plane and vice versa). Workspace RBAC applies: Viewer reads, Contributor writes. - Managed-folder enforcement (
onelake-api-parity.md): HEAD-only at account/workspace level; item root + first level protected from create/rename/delete;setAccessControl-class params rejected; banned headers ignored + echoed viax-ms-rejected-headers; canned$superuser/---------permission response headers. - Name- and GUID-addressing resolve to the same workspace/item.
- e2e: full write flow (create → append ×2 → flush) via GUID addressing, read back via name addressing; listings; RBAC walls; managed-folder rejections — against real entra-minted Storage tokens.
- Shortcuts: OneLake-to-OneLake symlinks — create/list/get/delete
(
internal/api/shortcuts.go); data-plane read/HEAD resolution through the target with target-side RBAC (the trusted-workspace-access path: a read through a shortcut is authorized against the TARGET workspace); ADLS Gen2/Amazon S3 external read-through via Connections; Dataverse 501; dangling target 404; self-cycle rejected. Store + API + OneLake resolution and container-network reads tested. - e2e: the real Azure Blob SDK (
azure-storage-blob) round-trips through the emulator —e2e/adls-sdk(3-OS): uploads a pyarrow Parquet, downloads it byte-identical (found + fixed thex-ms-rangegap), lists blobs, DFS sees the same file. - e2e: the real
azcopybinary transfers through the emulator —e2e/azcopy(Linux): multi-block upload (Put Block + Put Block List), byte-identical download, DFS sees the same object. Auth is a forged Storage token handed to azcopy in its static-token mode (TokenStore), since azcopy’s own MSAL flow validates the authority against public AAD.
R — Real compute (PySpark, Delta, warehouse)
Section titled “R — Real compute (PySpark, Delta, warehouse)”Designed in 14-real-compute.md: attach real engines below the emulated planes — never fake results. Lives in this repo (storage completeness + e2e harnesses + compose sidecars); only a future TDS-FedAuth proxy would be a separate sibling.
- R0 — OneLake storage completeness: the Blob-endpoint dialect
(
internal/onelake/blob.go— Put Blob / staged blocks / Copy / List Blobs XML paging, reached viaonelake.blob.*or the account-prefixed/onelake/{ws}/…path), Range reads (206) on both surfaces, ETags + put-if-absent conditional writes (Delta_delta_logatomicity), DFS rename (x-ms-rename-source), ETag/Last-Modified on every path. e2e A1 (e2e/delta-rs, CI): realdeltalakewrites v0 → reads back → appends v1, and the same files list through the DFS surface. Hardened with a concurrent-commit race test (24 goroutines race one_delta_logfile; exactly one wins — the mechanism-level atomicity oracle,-race-clean) andx-ms-rangesupport (found by the ADLS SDK). - R1+R2 (merged) — containerized Spark, its own focused runway.
Decision: R1 (in-process PySpark via ABFS) is folded into R2’s Spark
sidecar rather than pursued standalone. The Hadoop JVM ABFS driver
derives its endpoint from the URI authority and takes no host/port
override — unlike delta-rs’s
azure_endpoint(A1 ✅) or the Blob SDK’saccount_url(ADLS-SDK e2e ✅), both of which redirect cleanly. Only a container network whereonelake.dfs.fabric.microsoft.comresolves to the emulator solves that structurally — and that’s the R2 shape anyway, the production-faithful path, and the one that gives Windows users a real story (JVM stays in a Linux container; the client is thin). A separate weight class (a multi-hundred-MB Spark image + Docker orchestration), so it gets its own session rather than blocking the pure-wheel oracle work.- A2 — real JVM Spark + delta-spark write and read a Delta table
(2 commits) via the ABFS driver onto the OneLake plane,
e2e/spark(compose: entra + fabric-from-source + Spark; a custom token provider bridges ABFS’s v1resource=to entra’s v2scope=; a seeded storage resource app resolves the audience). Found + fixed a real bug: ABFS sends append/flush asPUT ?action=…, notPATCH— the flush PUT (empty body) was truncating every file to zero, silently corrupting Delta commits (regression-tested). Linux-only CI. - B (Livy passthrough contract) — the documented endpoint
(
…/lakehouses/{id}/livyapi/versions/2023-12-01/{sessions,batches}/…) is a bearer-validated, RBAC-gated reverse proxy (internal/api/livy.go) to a real Apache Livy backend set via--spark-livy-url/FABRIC_SPARK_LIVY_URL. Session-create and job-submit need Contributor; status reads need Viewer; unknown lakehouse 404s. Unset → honest 501. Unit-tested (path rewrite, RBAC matrix, 501, lakehouse) + a server e2e (real entra token → auth → RBAC → proxy → backend). - B (high-concurrency Livy sessions) — Fabric’s own layer on top
of the Livy contract (
highConcurrencySessions, current — it gained HC support in 2026): the emulator implements the packing manager directly (internal/api/livy_hc.go), since a vanilla Livy server has no REPL/HC concept.sessionTagpacks REPLs into a shared underlying Livy session, capped at 5 REPLs/session with spill-to-new-session; acquire is non-idempotent (same tag → distinct HC ids, sharedsessionId); acquire/get/delete are pure control-plane (no Spark); a REPL’s statements run on real Spark via the native agent (--spark-agent-url) — or proxy to an external Livy backend (--spark-livy-url), honest 501 without either. Unit-tested (packing, cap, spill, slot-reuse-after-release, RBAC) + a server e2e proving the HC routes win over the classic catch-all on the real mux, race-clean. - B (native Livy sessions on real Spark) — Apache Livy is retired
to the Apache Attic (no maintained image to bundle for the protocol), so
rather than proxy it, the emulator terminates the Livy contract itself
and drives a Spark statement-executor agent via
--spark-agent-url/FABRIC_SPARK_AGENT_URL(internal/api/livy_native.go,python/spark_agent/agent.py): sessions, statements, and batches run on real Spark, session state persists across statements, and HC REPLs each get their own agent namespace — so the 5-REPL model is real end to end (e2e/livy, containerized).--spark-livy-urlstill reverse-proxies an external Livy backend if a user brings one; unset → honest 501.
- A2 — real JVM Spark + delta-spark write and read a Delta table
(2 commits) via the ABFS driver onto the OneLake plane,
- R3 (SQL analytics endpoint — DuckDB) — real DuckDB runs SQL
(aggregation, join, filter) over Delta tables in the OneLake plane,
e2e/duckdb(3-OS): delta-rs writes two Delta tables into OneLake, DuckDB queries them and the results match — the lakehouse↔warehouse SQL interop, cross-engine. (DuckDB embeds via CGO, which the pure-Go distroless build forbids, so the SQL engine runs in the e2e, not the binary; the storage read is byte-proven by the delta-rs e2e.)- R3 (T-SQL / TDS warehouse) — T1–T5 done. Designed in
16-warehouse-tds.md. A pure-Go TDS endpoint
(
internal/tds,-sql-tds-addr) terminates Entra FedAuth (token validated vs entra’s JWKS,database.windows.netaudience); T3 reflects a lakehouse’s Delta into a real SQL Server sidecar (-warehouse-sql-url) — readTables/<t>Delta in pure Go (internal/warehouse),CREATE+INSERTthe rows;SELECTmatches DuckDB (R3/C1), the cross-engine oracle. T4 routes each item to its own database (Lakehouse read-only / Warehouse read-write, isolated), enforces RBAC→SQL permissions, and deliversinformation_schemaparity, native per-column type fidelity, and connect-by-name. T5 replaces the per-batch relay with a session splice (internal/tds/splice.go,client.go): after terminating FedAuth, the client’s post-login session is byte-forwarded to a real per-item SQL Server connection, so the engine emits every token natively (transactions, RPCs, prepared statements). That unlocks a second, independent driver family — Microsoft ODBC Driver 18 — so Microsoft’s real dbt-fabric adapter passesdebug→seed→run→testend to end (e2e/dbt-fabric), alongsidego-mssqldb. Not PolyBase: a spike proved SQL Server reading OneLake Delta directly is a dead-end on Linux (the object-storage connector components aren’t shipped), so reflection is the permanent design. (Front in this repo; engine a compose sidecar.)
- R3 (T-SQL / TDS warehouse) — T1–T5 done. Designed in
16-warehouse-tds.md. A pure-Go TDS endpoint
(
- R4 (notebook developer loop) — a functional
notebookutils/mssparkutilsshim (python/notebookutils, stdlib-only) that makes real Fabric notebook code run unchanged against the emulator family:fsover OneLake (create→append→flush, ranged reads, ls, cp — abfss URIs and lakehouse-relative paths),credentials.getTokenfor any audience,credentials.getSecretbrokered through the real azure-keyvault-emulator, thelakehousecontrol plane,runtime.context, andnotebook.runvia the jobs API. Proven bye2e/notebookutils(3-OS): entra + fabric + azure-keyvault up, a real notebook drives every module to a PASS. Designed in 14-real-compute.md (Track D).- R4 (real notebook cell execution) — a RunNotebook job is parsed
by the emulator (real Go parser,
internal/notebook:notebook-content.py→ ordered code cells, magics/markdown handled) and executed by real Spark:e2e/notebook-runpublishes a Fabric notebook, real JVM Spark runs its cells against the OneLake plane (a Delta table actually lands), and the engine reports per-cell results + exit value back so the job’s terminal status reflects the real run — not the clock. The parse/record/report contract is Go-side + unit-tested; the compute is real Spark (Linux-only e2e, reusing the spark-a2 image). Without an engine the cells are honestly “parsed, Pending”. - R4a (default-lakehouse session binding) — notebook metadata is
resolved and validated; Sail and JVM witnesses run unqualified table APIs
against the attached lakehouse’s OneLake
Tables/directory. - R4b (VS Code Fabric-extension compatibility) — the private
shared-backend/MWC protocol used by Microsoft’s Fabric Data Engineering
extension 1.18.1 is implemented for workspace/artifact discovery,
Notebook/SparkJobDefinition/Environment authoring, notebook content with
ETag conflicts, notebook resources, Spark-job history/cancel, and
lakehouse table discovery.
e2e/vscode-extensionreplays the pinned extension contract through the realapi.powerbi.comhost alias with an entra-emulator Power BI-audience token. Interactive Jupyter kernel websockets and table preview remain out of scope; notebook execution is available through the existing jobs/Livy/Sail surfaces.
- R4 (real notebook cell execution) — a RunNotebook job is parsed
by the emulator (real Go parser,
- R5 (DataPipeline interpreter) — a real, pure-Go interpreter
(
internal/pipeline) for Fabric/ADF Data Pipeline definitions: the full expression language (a faithful subset —pipeline(),variables(),activity(),item(), and the string/logic/math/array function library with ADF-loose coercions), control flow (IfCondition, ForEach, Until, Switch, Filter, Fail), variables (Set/Append),dependsOnwith all four dependency conditions (Succeeded/Failed/Completed/Skipped), Invoke pipeline (ExecutePipeline — real recursive interpretation of a referenced DataPipeline, with parameter flow,waitOnCompletion, and a cycle guard), and per-activity policy (retry re-runs a failed activity and recordsretryAttempt; timeout fails an over-running attempt — deterministic, no real sleeping). Wired into the jobs API: aPOST …/jobs/instances?jobType=Pipelineon a DataPipeline item executes the definition now, a pipeline failure sets the job’s terminal status, and…/jobs/instances/{jid}/queryactivityrunsreturns the per-activity run detail. The notebook leaf activity (TridentNotebook) chains a real RunNotebook job — pipeline → jobs → notebook, end to end. Proven by interpreter unit tests, API-level job tests, and a server e2e (real entra token → auth → RBAC → interpreter → queryactivityruns). Coverage floor held at ≥90%. A malformed expression fails the activity (recovered), never the server.- R5 (real data-plane leaf activities) — leaves that can run for
real, hermetically where possible: Copy moves real bytes
OneLake→OneLake (a file, or a directory subtree, with expression-resolved
{workspaceId?, itemId, path}locations); Lookup reads real rows from a CSV/JSON/Parquet file or a lakehouse Delta table (Tables/<name>, auto-detected) and feeds@activity(…).output; GetMetadata stats a real path (exists/itemType/size/lastModified/childItems); Script/SqlServerStoredProcedure run real T-SQL against a Warehouse/SQLDatabase item’s own SQL Server database (Track C’s backend), targeted the same{workspaceId?, itemId}way as Copy/Lookup. Web, WebHook, REST, Salesforce and Custom (Azure Batch) run for real (Web and Custom can be refused withFABRIC_WEB_ACTIVITY=stub/FABRIC_CUSTOM_ACTIVITY=off). External stores on Copy stay refused by name. - R5 (Apache Airflow + Dataflow Gen2 boundary) —
ApacheAirflowJobitem/file APIs sync Python DAGs into an attached Airflow 2.10.5/Python 3.12 sidecar, unpause and trigger them through the upstream REST API, then derive Fabric job completion from the real DAG run.e2e/airflowproves real scheduler + executor + task execution with a shared DAG volume. The sidecar is opt-in through--airflow-urland--airflow-dag-dir; an unattached engine fails explicitly.Dataflowmanagement/definition round-trip is supported, while Refresh/Publish and in-pipeline Dataflow execution fail withDataflowEngineNotImplemented: Power Query M is proprietary, so there is no engine to attach and no result is faked. Designed in 14-real-compute.md (Track E).
- R5 (real data-plane leaf activities) — leaves that can run for
real, hermetically where possible: Copy moves real bytes
OneLake→OneLake (a file, or a directory subtree, with expression-resolved
Cross-cutting (throughout)
Section titled “Cross-cutting (throughout)”- Svelte portal: dashboard / workspaces (items, role assignments, git
status drill-down) / operations / clock / fault injection / workspace
identities — served at
/on the control-plane origin, reading state through unauthenticated/_emulator/portal/*endpoints (the /v1 contract stays bearer-only).go:embed all:dist+ committeddist+ CI drift guard; 21 Vitest unit tests. - Starlight docs site on GitHub Pages (this
/docs= source of truth, synced bywebsite/scripts/sync-docs.mjs; pinned Astro Starlight; deploys viadocs-site.yml) — live at https://calvinchengx.github.io/fabric-emulator/. - GoReleaser: binaries + distroless Docker (GHCR, HEALTHCHECK via the new
healthchecksubcommand) + Homebrew cask + winget (both self-skip without their tokens);versionstamped via ldflags. Channels go live at the firstv*tag. - Playwright headless mount smoke (catch builds-but-doesn’t-mount) — in the portal CI job, with the vite
resolve.conditionsfix baked in. - Coverage parity with entra-emulator (≥ 70% per package): every package 77–100% from its own tests; 91.6% total plain / 93.5% cross-package (CI floor 90%).
- Connection credentials:
credentialDetails.credentialType(Fabric’s ten: Anonymous / Basic / Key / KeyPair / OAuth2 / ServicePrincipal / SharedAccessSignature / Windows / WindowsWithoutImpersonation / WorkspaceIdentity) with write-only secrets, SP validation against entra at create (skipTestConnectionbypass), and the WorkspaceIdentity kind gated on a provisioned identity. - Vault-backed credentials: a
KeyVaultSecretReferenceonkeyReference/passwordReference/tokenReference/servicePrincipalSecretReferenceresolves the secret via the workspace identity’s vault-audience token against azure-keyvault-emulator —workspace identity → entra token → vault secret → connection, offline (internal/akv; only the pointer is stored, never the value).
S — Engine swap: LakeSail replaces JVM Spark
Section titled “S — Engine swap: LakeSail replaces JVM Spark”Designed in 20-lakesail-engine.md: Sail (Rust Spark Connect) becomes the engine behind every compute surface — PySpark with no JVM.
- S0 — proof:
e2e/sail(CIsail, Linux): real PySpark Connect client → Sail → Delta write/SQL/append through the OneLake Blob surface (az:// + endpoint override, the delta-rs recipe), entra client-credentials minted by thedocker/saillauncher. Conditional-PUT Delta commits ride the R0 contract unchanged. - S4 (pulled forward) — user-facing compose: the default
docker-compose.override.ymland the explicitdocker-compose.compute.ymlboth run Sail + the thin Connect agent (JVMapache/sparkimage dropped); validated end-to-end over the compose network incl. self-signed-TLS storage writes. Fabric builds from the tree until a post-Blob-surface release is tagged. - S1 —
e2e/livy+e2e/dbt-fabricsparkcomposes swapped to Sail (apache/spark:3.5.3gone; agents are Spark Connect clients; under Sailscis a guide-rail stub, and the local-relation limit override keepscreateDataFrameworking for user code). - S2 —
e2e/notebook-runrunner connects viaSPARK_REMOTE; the notebook fixture executes unmodified (abfs:// URLs andcreateDataFrameincluded); JVM image build dropped. - S3 —
e2e/spark(A2) reborn on Sail with the same production-shapedabfs://URLs;EntraTokenProvider.java+ the JVM Dockerfile deleted. The default path has no JVM; the opt-in overlay (docker-compose.spark-jvm.yml/make up-jvm) later brought JVM Spark back for RDD, checkpointed streaming, and Java/Scala UDFs. Tradeoff accepted on the default: the Hadoop-ABFS driver witness lives on that overlay, not on Sail.
After S — shipped on the same contract
Section titled “After S — shipped on the same contract”These landed after the Sail swap and are CI-verified. They are not a new phase letter; they extend surfaces the earlier phases already named.
- Eventstream — Apache Kafka KRaft sidecar (
--profile eventstream), Fabric notebook API (format("kafka")+eventstream.*), Custom HTTP produce. Lakehouse destination appends produce payloads as Delta; Reflex destination firesMicrosoft.Fabric.Eventstream.EventReceivedas a realEventTriggeredjob. Eventhouse destination ingests via Kusto.create-merge+.ingest inline(direct, not Fabric streaming ingest). Operators (Filter, GroupBy, tumbling Window) run on the produce batch. 51-eventstream-kafka.md. - Custom activity on by default — Azure Batch
commandruns on the Spark agent;FABRIC_CUSTOM_ACTIVITY=offrestores the refusal. - Fabric Core MCP —
POST /v1/mcp/core, unmodified PythonmcpSDK in CI. - Optional
msmdsrvDAX oracle —FABRIC_DAX_URLrelaysexecuteQueriesto a pump in front of Desktop’s engine on a machine you own. Not a compose default. 52-msmdsrv-hosts.md.
Sequencing note
Section titled “Sequencing note”Build the LRO engine before anything that mutates — every workspace/item/git
call returns through it, so getting 202 → poll → terminal right once makes all
later endpoints trivial. P2’s entra-side dependency (#16) has already shipped,
so phase order is a pure prioritization choice, not a blocking one.