E2E matrix
What CI proves on every push, and what’s queued. The bar, inherited from entra-emulator’s SDK matrix: real clients, unmodified, against the emulator — because driving Microsoft’s actual tools catches fidelity gaps spec-reading cannot (see what fabric-cicd caught).
Verified on every push
Section titled “Verified on every push”Go integration tests start a real entra-emulator in-process and drive the full HTTP surface; the remaining suites drive real third-party clients and engines against the running emulator.
Default compute tier: every Spark row below (
spark-a2,livy-native,dbt-fabricspark,notebook-run) runs on LakeSail’s Sail (Rust Spark Connect, no JVM). This proves the Spark Connect/DataFrame subset, not full Microsoft Fabric Runtime parity — see 20-lakesail-engine.md.
| Suite | Client | Proves | Where |
|---|---|---|---|
| Token handshake | in-process entra-emulator | real client-credentials token (Fabric aud) → JWKS validation → full workspace/RBAC/item/LRO flow over HTTP | Go integration tests (CI test, Linux + macOS + Windows) |
| Git round-trip | Go HTTP | two-workspace commit→update, definitions intact, logical ids preserved | Go integration tests |
| Identity handshake | in-process entra-emulator | provision → entra mints for the identity → token passes fabric RBAC → deprovision revokes → delete cascades | Go integration tests |
| OneLake | Go HTTP + real entra Storage tokens | create/append/flush/read via GUID + name addressing, listings, RBAC walls, managed-folder rejections | Go integration tests |
| fabric-cicd | Microsoft’s real Python tool (v1.3.x) | publish_all_items publishes a notebook, a Variable Library (nested valueSets/ parts) and a DataPipeline that references it; parts round-trip byte-for-byte, and the emulator then RUNS the published pipeline — which fails by construction unless the library variable resolved, so a green run is evidence rather than absence of error. The value-set switch is exercised through the same published definitions. Found the /VariableLibraries/ casing divergence that no spec-reading would have | e2e/fabric-cicd/run.py (CI fabric-cicd, 3-OS) |
Fabric CLI (fab) | Microsoft’s own ms-fabric-cli 1.6.1 | service-principal login through entra-emulator’s MSAL flow, then mkdir/ls/get/exists/rm over Notebook/SemanticModel/Report/DataPipeline/Lakehouse and the raw fab api passthrough. Drives the deployment-pipeline surface in the call order of Microsoft’s own DeploymentPipelines-DeployAll.ps1 (list → stages → assignWorkspace → deploy → poll → result), asserting a re-deploy reports Updated rather than duplicating the paired item and that the history keeps both deploys newest-first with the note intact. Also the list continuationToken contract driven by a client we did not write: with FABRIC_LIST_PAGE_SIZE=2 a plain list pages unprompted, fab carries the opaque token across ≥3 pages with no item returned twice or lost, and continuationUri comes back absolute carrying that same token | e2e/fabric-cli/run.py (CI fabric-cli, Linux, containerized) |
Azure CLI (az rest) | Microsoft’s own az CLI (image mcr.microsoft.com/azure-cli, unmodified) | service-principal login through entra-emulator, then az rest against Fabric REST and the Power BI admin activity log. Witnesses already-green rows a packaged Microsoft client actually speaks: capacities list/assign/unassign, folders, workspace RBAC, job schedules and instances, queryactivityruns, tenant settings, admin workspaces/items, capacity overrides, governance domains, sensitivity labels, activityevents, workspace-identity provision/deprovision, report definition round-trip, and the deleted-name 409 isRetriable hold. Also the git integration round trip (connect, initializeConnection’s required action, commit, a second workspace pulling through updateFromGit, disconnect) and CopyJob, whose bytes are seeded and read back over OneLake with the Storage audience rather than inferred from a Completed status. Does not claim pipeline activity interpreters (see the row below), races, or emulator-only headers | e2e/az-rest/run.py (CI az-rest, Linux, containerized) |
| Azure CLI — pipeline activities | the same unmodified az CLI, in its own job | creates, defines and runs real pipelines, then judges each activity by the data rather than the reported status: Delete (target gone, sibling intact), GetMetadata (size against the bytes written), Lookup (rows read, and the value arriving in a downstream SetVariable), Validation (present data passes, absent data fails the job), ExecutePipeline (the child’s byte move, not the parent’s status), ForEach (3 iterations, not 1), control flow (the taken branch runs and the skipped one does not), retry policy (one record carrying retryAttempt: 2), and Web/WebHook (a real HTTP call, then a park that only the callback releases). Separate from az-rest so one container cannot take two dozen parity rows red at once | e2e/pipeline-activities/run.py (CI pipeline-activities, Linux, containerized) |
Terraform (microsoft/fabric) | Microsoft’s own Terraform provider (image hashicorp/terraform, unmodified provider from the registry) | service-principal auth through entra-emulator, then terraform apply / destroy of fabric_workspace, nested fabric_folder, fabric_lakehouse, fabric_workspace_role_assignment, and data.fabric_capacity against the seeded F64. Witnesses already-green control-plane rows; does not claim ARM (azurerm), Spark, OneLake bytes, git, or workspace identity | e2e/terraform-fabric/run.py (CI terraform-fabric, Linux, containerized) |
| Deployment pipelines from PowerShell | real PowerShell 7.4.18 + Microsoft’s Az.Accounts | a second, independent MSAL implementation on the same surface — .NET via Connect-AzAccount -ServicePrincipal + Get-AzAccessToken, where the fab row above reaches it from Python, so a token that satisfies only one MSAL cannot pass both. Then DeployAll.ps1’s exact REST sequence: three default stages with Development first, deploy answers 202 with x-ms-operation-id, polling honours Retry-After, the result’s targetItemId equals the id the target workspace actually lists, and a second deploy reports Updated with the target still holding exactly one item | e2e/deployment-pipelines/run.py (CI deployment-pipelines-ps, Linux, containerized) |
| Delta write/read (A1) | real deltalake (delta-rs) | a real engine writes/reads a Delta table through the OneLake Blob surface with an entra Storage token — Range reads + the _delta_log put-if-absent commit primitive | e2e/delta-rs/run.py (CI delta-rs, 3-OS) |
| Sail / Spark Connect (S0) | LakeSail’s real sail server + pyspark-client | a Rust Spark-Connect engine (no JVM) writes/reads Delta and runs SQL through the OneLake Blob surface — same object_store contract as delta-rs, entra-authenticated via the launcher mint | e2e/sail/run.py (CI sail, Linux) |
| fabric_target toggle (T0) | the fabric-target package | one FABRIC_TARGET switch resolves endpoints + credentials: seeded TokenCredential mints per-scope, session drives workspace-by-name → item → LRO, real-mode guards (workspace scope, az-login-or-SP, destructive gate) enforce; plus the T1 conformance suite (same 7 tests run against real Fabric via the secret-gated real-fabric workflow) | e2e/fabric-target/run.py (CI fabric-target, 3-OS) |
| Real-Time Intelligence (KQL) | Microsoft’s own kustainer KQL engine + Microsoft’s azure-kusto-data SDK + raw Kusto REST | the eventhouse’s published queryServiceUri really executes KQL: create a table, ingest rows two documented ways, query the values back (summarize, datetime filter), and prove per-KQL-database isolation; Kusto-audience bearer + RBAC enforced, engine database naming never leaks | e2e/rti/run.py (CI rti, Linux amd64 only — the engine needs AVX2 and Microsoft documents ARM as unsupported) |
| Governance (OpenMetadata) | real OpenMetadata 1.13.2 (Postgres) + delta-rs | catalogs Delta schemas, shortcut lineage, and an executed pipeline Copy edge (lake.orders → curated.orders_copy) returned by OM’s graph API; idempotent on re-ingest | e2e/governance/run.py (CI governance, Linux) |
| Portal terminal pane | real Chromium (the portal’s own Playwright) + real ttyd | the Flow view’s terminal is a working shell, not a widget. Six checks: the emulator dials ttyd across the compose network (so the toggle appearing is the proof), the iframe loads ttyd from the portal’s origin, xterm attaches, a typed echo …-$((6*7)) → …-42 proves a real shell and ttyd -W, a wrong token gets 401, and a plain portal route still answers JSON while the pane is live — the regression where the proxy spliced every request, not only websocket upgrades. Boots the documented way (--profile terminal + docker-compose.terminal.yml), so a URL set without its service is caught | e2e/portal-terminal/run.py (CI portal-terminal, Linux) |
| Governance SSO | real OpenMetadata + entra-emulator | OM’s authenticator is entra: a forged user token is accepted by OM’s API and a broken-signature token is refused — the catalog inside the family trust chain | e2e/governance/sso.py (CI governance-sso, Linux) |
| ADLS SDK | Microsoft’s real azure-storage-blob | Parquet upload → byte-identical download (exercising x-ms-range), list_blobs, and the DFS surface sees the same file | e2e/adls-sdk/run.py (CI adls-sdk, 3-OS) |
| Fabric Core MCP | official Python mcp SDK (Streamable HTTP) | initialize, tools/list of the published Core MCP names, the get-started prompts (list/create workspace, create a Lakehouse, search_catalog), then the rest of the published Core list that does not execute notebooks or write lakehouse tables — folders, roles, capacities, bulk move, get_knowledge as plain text | e2e/mcp-core/run.py (CI mcp-core, Linux) |
| azcopy | Microsoft’s real azcopy binary | multi-block upload (Put Block + Put Block List) → byte-identical download, and the DFS surface sees the same object | e2e/azcopy/run.py (CI azcopy, Linux) |
| Spark API on Sail (A2) | real PySpark (Spark Connect client) + LakeSail’s sail | writes/reads Delta over production-shaped abfs://…@onelake.dfs… URLs — engine is Rust, no JVM | e2e/spark/run.py (CI spark-a2, Linux, containerized) |
| Eventstream on Sail | real PySpark Connect + Apache Kafka KRaft | Fabric notebook API (format("kafka") + eventstream.* + foreachBatch) against a real topic: Kafka schema, row count, unknown IDs fail, never rate. Same job binds a Lakehouse destination and a Reflex destination: asserts Tables/<name>/_delta_log content and five EventTriggered pipeline jobs after Custom HTTP produce. Consume via the emulator (LocalRelation); foreachBatch runs in the client. Same job also reads OSS format("kafka") + bootstrap/subscribe so CAST(value) runs on Sail. Pattern / assign / SASL PLAIN / kafka sink are unit-tested on the wrap, not this job | e2e/eventstream/run.py (CI eventstream-sail, Linux, containerized) |
| Native Livy | real Livy REST client + Sail | emulator terminates the Livy protocol itself and drives a statement agent — session + PySpark statements computed by a real engine (Sail, no JVM), no Apache Livy server. Also the Delta maintenance witness: OPTIMIZE/VACUUM/Change Data Feed run through delta-rs against a real abfss://…onelake… table, with a negative control (an invalid Storage bearer must be refused, so a pass cannot mean OneLake skipped the check) | e2e/livy/run.py (CI livy-native, Linux) |
| Fabric VS Code extension contract | Microsoft Fabric Data Engineering extension 1.18.1 route fixture | Power BI discovery/auth, workspace/artifact authoring, notebook content/resources with ETags, and host redirection through api.powerbi.com | e2e/vscode-extension/run.py (CI vscode-extension, Linux) |
| Apache Airflow Job | real Apache Airflow 2.10.5/Python 3.12 | uploaded DAG discovery, scheduler/executor task run, REST polling, and the resulting Fabric job terminal state | e2e/airflow/run.py (CI airflow, Linux) |
| Data science loop | PySpark/Sail + Direct Lake DAX + MLflow 3 + dbt-duckdb Delta plugin | one Spark-written OneLake Delta table is queried by DAX, tracked as typed MLflow experiment/model items with mirrored artifacts, then built and tested by dbt | e2e/data-science-loop/run.py (CI data-science-loop, Linux) |
| Semantic model DAX (executeQueries) | stdlib HTTP + real entra Power BI-audience tokens | the golden TMSL model published as a SemanticModel item, then the consumer’s path rather than the publisher’s: /v1.0/myorg/groups/{ws}/datasets lists the published id inside the OData wrapper the swagger defines (an LRO handle proves nothing about discoverability), that list 401s a Fabric-audience token, and three golden DAX queries through executeQueries match a hand-computed oracle row for row. Plus the honest negatives for an inline-data model: datasources is [], refreshes is 400 rather than a Completed that would tell a caller their numbers were refreshed when nothing was re-read, and isRefreshable=false agrees with that refusal instead of contradicting it | e2e/semantic-model/run.py (CI semantic-model, 3-OS) |
| Great Expectations | real great_expectations 0.18.22 + pandas | the SemPy/GX tutorial’s own suites run against executeQueries results, reproducing the tutorial’s pass/fail split: Store row-count-in-range and 5-digit-zip regex pass, TotalUnits >= 50000 passes, and the YoY-ratio asset fails — with 1.8 asserted present in the unexpected values, so a green run cannot come from a suite that validated nothing. DAX is single-sourced from the semantic-model golden fixture, so the two rows cannot drift apart | e2e/great-expectations/run.py (CI great-expectations, 3-OS) |
| dbt (fabric-spark) | Microsoft’s real dbt-fabricspark adapter | a dbt project (debug → seed → run → test) over the Fabric REST + Livy HC surface, models computed by Sail (no JVM) | e2e/dbt-fabricspark/run.py (CI dbt-fabricspark, Linux) |
| dbt (fabric) via mssql-python | Microsoft’s real dbt-fabric adapter 1.11 + mssql-python | a dbt project (debug → seed → run → test) over the TDS warehouse surface through mssql-python + FedAuth (byte-spliced to a real SQL Server) — a third independent TDS driver family (go-mssqldb and pyodbc/ODBC 18 remain covered by warehouse-tds and medallion warmup) | e2e/dbt-fabric/run.py (CI dbt-fabric, Linux) |
| Medallion tutorial, end to end | real dbt-fabric + Microsoft ODBC Driver 18 + LakeSail Sail + delta-rs | the whole analytics loop driven through the emulator’s own surfaces, not around them: a Key Vault secret resolved through a Key credential’s keyReference into an AzureKeyVault connection (never returned on read); a real DataPipeline whose Copy activity the emulator itself executes (Fabric’s DelimitedTextSource/LakehouseTableSink payload → Delta, reporting rowsCopied) and whose Notebook activity starts a genuine run that Sail executes over Spark Connect, writing Delta and reporting its read/write set; silver → gold with dbt over TDS; a DAX query through executeQueries; plus the inverse assertion that the gold DQ tests fail on poisoned silver, and a lineage assertion that the graph carries both a Copy edge (recorded by the emulator) and a Notebook edge (reported by the engine) — neither inferred from user code. Runs examples/medallion-pyspark unmodified — the witness for 28-tutorial-end-to-end.md | e2e/medallion/run.py (CI medallion, Linux) |
| Advanced medallion — three sources resolved | same clients, plus a nested-JSON and a Parquet CDC feed | what a second and third source system force, which a one-source pipeline never has to face: no key spans all three (POS↔Web on email, POS↔ERP on phone, ERP↔Web nothing), so identity is resolved transitively — an ERP account reaches a Web account only through POS — and the step that resolves also names the three cohorts it cannot place. SCD2 turns the ERP change log into a dimension with history; the star joins both selling channels in the Warehouse; the ODCS contracts run as executable gates at every layer and are proved able to fail. Runs examples/medallion-advanced-pyspark unmodified | e2e/medallion-advanced/run.py (CI medallion matrix, Linux) |
| Medallion engine pair | Microsoft’s dbt-fabricspark over Livy HC, Sail behind it | the same two pipelines with silver built declaratively instead of in PySpark, so a difference between them is attributable to the engine and not the code around it — enforced by a parity check that fails the build if the pair diverges anywhere but silver, and by a compare job asserting both silvers agree row for row | e2e/medallion-dbt-fabricspark/, e2e/medallion-advanced-dbt-fabricspark/ (CI medallion matrix + medallion-compare) |
| DuckDB SQL | real DuckDB | SQL (aggregation, join, filter) over Delta tables in the OneLake plane — the lakehouse SQL-analytics-endpoint semantics | e2e/duckdb/run.py (CI duckdb, 3-OS) |
| notebookutils (+ T2 target unit tests) | real Fabric notebook | the functional notebookutils shim: fs over OneLake, credential tokens, Key Vault secret brokering, lakehouse control plane, notebook.run; plus python/tests asserting the shim’s emulator-vs-real resolution (endpoints, TLS, DefaultAzureCredential, no seed leakage) | e2e/notebookutils/run.py (CI notebookutils, 3-OS) |
| Notebook + SJD + Environment | Sail and Spark 3.5/Delta 3.2 JVM | attached lakehouse metadata binds unqualified table APIs to OneLake; Environment requirements/config apply; SJD source+args execute and report; Sail rejects JAR requirements while JVM exposes its dependency surface | e2e/notebook-run/{run.py,run-jvm.py} (CI notebook-run, scheduled JVM oracle) |
| Notebook, no runner attached | Sail behind the published spark-agent | a Notebook item is published and a RunNotebook job submitted by a client that holds no Spark session and reports nothing; the emulator drives the agent, so the job reaches Completed, mssparkutils.notebook.exit stops the run and carries its value, the cell after the exit stays Pending, and the Delta table is read back out of OneLake over plain HTTP. The stack is only published services — this is the path a consumer with no clone can walk | e2e/notebook-driven/run.py (CI notebook-driven, Linux) |
| Power BI Desktop reads what we generate | Microsoft Power BI Desktop on windows-latest | a .pbix is built from this platform’s TMSL, opened by Desktop itself, and its Analysis Services instance queried over ADOMD.NET — the fixture DAX must match executeQueries bit-for-bit. Stages are reported separately, so “never hosted Analysis Services” and “ran and disagreed” stay distinguishable. A spike promoted on a measured 5/5 pass rate, not on one green run; the parsing and comparison logic is unit-tested off Windows | e2e/pbix-desktop/run.py (CI PBIX Desktop probe, weekly + manual) |
| Spark engine matrix is current | Sail, Sail + delta-rs, and Apache Spark JVM | 23 capability probes run against all three engines, one probe per capability, and docs/engine-matrix.md is regenerated and diffed — so a claim like “SQL VERSION AS OF is a Sail gap” cannot outlive the Sail upgrade that closed it, and no cell can bundle two capabilities behind one verdict that hides a partial pass | e2e/engine-matrix/run.py (CI engine-matrix, Linux, git diff --exit-code docs/engine-matrix.md) |
| Framework conformance matrix is current | generated from e2e/conformance/out/*.json | every required/control cell of 38-framework-conformance.md has a verdict, every ❌ a pointer; docs/conformance-matrix.md is regenerated and diffed. Contract 4 is live: unqualified saveAsTable("events") (sail, jvm) and CREATE+INSERT (warehouse) through the emulator path, confirmed by a OneLake DFS listing or a fresh TDS connection — the engine that wrote is never the one that confirms | e2e/conformance/run.py --live (CI conformance-sail / conformance-jvm / conformance-warehouse, git diff --exit-code docs/conformance-matrix.md e2e/conformance/out/) |
| External shortcuts | containerized HTTP object stores | ADLS Gen2 and Amazon S3 shortcut definitions read real remote bytes through authenticated OneLake requests | e2e/external-shortcuts/run.py |
| S3 shortcut, SigV4 enforced | Amazon’s own boto3 + a real SeaweedFS S3 server | depth where the row above gives breadth: SeaweedFS runs with an identity config, so an AmazonS3 shortcut only resolves if the emulator signs the upstream request itself. boto3 puts the object, an unsigned GET must be refused (without which the read-through would demonstrate nothing), the OneLake ADLS surface returns the bytes byte-for-byte, and a second shortcut carrying a deliberately wrong secret key must be refused — a pass cannot mean the signature went unchecked | e2e/s3/run.py (CI external-s3, Linux, containerized) |
| ADLS Gen2 shortcut on Azurite | Microsoft’s own Azurite + azure-storage-blob | the same depth for the other kind, against Microsoft’s storage emulator instead of a hand-written stub: unauthenticated GET refused, read-through returns the blob under a real SAS, a tampered sig= refused, then create/append/flush through the shortcut producing bytes the SDK reads back straight from Azurite — the target, not the emulator — and a delete that removes them there too. Azurite implements Blob and not DFS, so the write protocol’s DFS-specific semantics are deliberately not claimed | e2e/azurite-shortcut/run.py (CI azurite-shortcut, Linux, containerized) |
| Warehouse TDS | real go-mssqldb + real SQL Server 2022 | entra-token connect, then DDL + DML + a GROUP BY relayed through the TDS endpoint — one of two independent TDS driver witnesses (the other: Microsoft ODBC Driver 18 via dbt-fabric above); plus the SQL Database → OneLake Delta mirror, the pipeline Script/SqlServerStoredProcedure activities over real HTTP + jobs, and an external-source MirroredDatabase mirror (seeded on a database reached independently of the emulator’s own per-item routing) | CI warehouse-tds (Linux) |
| ARM capacities consume | Microsoft’s azure-mgmt-fabric 1.1.0b1 | a capacity created on arm-emulator (Microsoft.Fabric/capacities) appears on Fabric GET /v1/capacities under the Fabric REST GUID ARM assigned at create; the seeded default stays. Sibling checkout if present, otherwise go install of the pinned arm-emulator release | e2e/arm-capacities/run.py (CI arm-capacities, 3-OS) |
Plus: coverage floor 90% (cross-package; currently ~90%), go vet, a
distroless container smoke (docker-smoke), the portal build + headless
render (portal), and the
docs site build on every
docs push.
Slower compatibility oracles
Section titled “Slower compatibility oracles”| Cadence | Suite | Proves |
|---|---|---|
| Weekly/manual | e2e/spark-jvm | Apache Spark 3.5.5 + Delta 3.2 + Java 11 batch Delta, Hadoop ABFS, RDD/SparkContext, JVM/JAR bridge, Structured Streaming, VACUUM and CDF |
| Weekly/manual | e2e/eventstream | Eventstream notebook API on JVM Spark + Apache Kafka KRaft: create item, bind Lakehouse destination, Custom produce, _delta_log content, format("kafka") + eventstream.* + foreachBatch, plus OSS bootstrap/subscribe, Kafka schema, unknown IDs fail, not rate |
| Weekly/manual | e2e/notebook-run/run-jvm.py | the same representative notebook used by Sail runs on the Fabric Runtime 1.3-aligned JVM baseline |
| Weekly + release | e2e/notebook-run/real_fabric.py | the representative DataFrame/SQL notebook publishes and completes in real Microsoft Fabric; secret-gated |
| Weekly/manual | e2e/xmla | Microsoft’s own ADOMD.NET (…AdomdClient.NetCore.retail.amd64) on Linux/.NET 8 aims at a host we name and its first call is captured off the wire: GET /powerbi/databases/v201606/workspaces?PreferClientRouting=true, User-Agent: ASClient/…, bearer taken from the connection string. Also asserts the powerbi:// form stays usable on Linux and the https://…/xmla and bare host:port forms stay Windows-only |
The Sail suite also asserts the negative boundary: RDD/SparkContext, Py4J,
spark.jars, streaming, OPTIMIZE, VACUUM, and ignored CDF options; it also
asserts one-winner/one-rejected concurrent overwrite behavior. A capability
change fails CI until this matrix is deliberately reclassified.
Queued (designed, not yet wired)
Section titled “Queued (designed, not yet wired)”None. Live write-landing (contract 4 on sail / jvm / warehouse) is in Verified above.
Running locally
Section titled “Running locally”go test ./... # everything in-process, no networkuv run --frozen --group fabric-cicd python e2e/fabric-cicd/run.pypython3 e2e/arm-capacities/run.py # sibling arm-emulator, or go install of the pinned releaseuv run --frozen --no-sync python e2e/vscode-extension/run.pyuv run --frozen --no-sync python e2e/airflow/run.pypython3 e2e/data-science-loop/run.pyuv run --frozen --no-sync python e2e/xmla/run.pyPython dependencies are defined in the root pyproject.toml and locked by
uv.lock; dependency-bearing E2Es use their named dependency group, while
stdlib-only Docker launchers use --no-sync. The commands are
deterministic: virtual clock, in-memory stores, seeded credentials.