Spark engine matrix — Sail vs JVM

Generated by e2e/engine-matrix/run.py. Do not edit by hand.

The same probe runs against both engines. A capability is never bundled with another: each row is one probe, and a failure records the engine’s own error message, because No table format found for: memory and unsupported extension node for streaming: DeltaWriteNode are different facts pointing at different fixes.

Rows where the engines differ are the honest content of the 🔴 default / 🟠 JVM overlay marks in parity.md.

Three columns, because engine and emulator are different things: Sail (engine) is the bare engine; Sail + delta-rs is what a user actually gets, since the Livy agent installs the delta-rs interception for every Sail session (20-lakesail-engine.md). The middle column runs the agent’s own module, not a re-implementation, so it cannot drift from the runtime it describes.

How to read a cell

A ✅ means this probe passed on this engine — nothing wider. A ❌ is not a verdict that the capability is impossible; it is the engine’s own error, kept verbatim because the message is the actionable part. Three distinct things hide behind ❌, and the footnotes say which is which:

1. A real gap — the engine cannot do it (sc / _jvm on Connect; checkpointed streaming on Sail; bare Sail has no kafka source).
2. A protocol limit — Spark Connect itself forbids it (sc, _jvm), so no upstream fix exists for any Connect client, Sail or Apache’s.
3. A harness artefact — the probe’s environment, not the engine. These are bugs in the table and get fixed rather than explained away: the Python UDF row was one, until the pyspark client pin was corrected to the version pysail is built against.

Read a ❌ next to a ✅ in the same row as “the emulator closes this gap”, and two ❌ with different messages as two different problems.

How the table is produced

Four rules, each of them a reaction to a way an earlier hand-written version misled:

• Generated, never edited. A hand-written row claimed streaming was broken on Sail long after Sail gained streaming. Prose drifts from reality silently; a generated file cannot. CI regenerates and diffs.
• One probe per capability, never bundled. Three capabilities once shared one verdict, which hid a partial pass. A row that covers two things can only report the worse of them.
• The error text is recorded, not summarised. No table format found for: memory and unsupported extension node for streaming: DeltaWriteNode point at different upstream fixes; “streaming sink fails” points at neither.
• The middle column imports the shipped module. It does not re-implement the interception. If the agent changes, the column moves with it — the table cannot claim a behaviour the runtime lacks.

What this table does not do: prove the OneLake path. The probes run against local table paths, so credentials are out of scope here and are witnessed separately by e2e/livy against abfss://…onelake…, with a negative control (a deliberately wrong bearer must be refused). Keeping those separate is deliberate — a capability probe that also needed storage auth would fail for two unrelated reasons and distinguish neither.

Which engine should I use?

Sail is the default, and should stay the default. This table only measures capability; it says nothing about the axis Sail was chosen for. Measured on the same 26 probes:

	Sail	Spark JVM
Image size	943 MB	2.1 GB
Run output	125 log lines	78,040 log lines
Wall clock	seconds	minutes

And the differences are fewer than the row count suggests:

• two (sc, spark._jvm) are Spark Connect protocol limits, not Sail choices — Apache Spark’s own Connect client cannot expose them either, so no upstream fix exists;
• one (MERGE at a local path) works on az://, the path the emulator actually uses.

The common notebook path — Delta write/append, both time-travel forms, createDataFrame, SQL, readStream — passes on both. Ordinary work sees no difference except speed.

Reach for the JVM overlay when your test touches Java/Scala UDFs / spark.jars, the RDD/_jvm surface, or a checkpointed streaming query (foreachBatch on an engine stream). Those are the ❌ rows that stay red in the middle column. Durable streaming sinks (delta / parquet / memory) land one announced micro-batch there — not a Fabric- shaped checkpointed query. OSS format("kafka") (source and sink) is the same class of wrap (driver consume/produce → Sail LocalRelation). OPTIMIZE/VACUUM, LOCATION-bearing CREATE TABLE, MERGE, DESCRIBE, Change Data Feed, JSON multiLine, and those sinks are closed on the Livy path. One flag:

docker compose -f docker-compose.yml -f docker-compose.override.yml \
               -f docker-compose.spark-jvm.yml up

Real Fabric Runtime 1.3 is JVM Spark 3.5.5 + Delta 3.2 + Java 11, so the overlay is the higher-fidelity engine. It is not the default because a 2.1 GB image and a minutes-long startup would cost every user speed to buy capabilities most tests never touch.

Capability	Sail (engine)	Sail + delta-rs (emulator)	Spark JVM
Delta write	✅	✅	✅
Delta append	✅	✅	✅
Time travel — option("versionAsOf")	✅	✅	✅
Time travel — SQL VERSION AS OF	✅	✅	✅
MERGE INTO a registered table at a local path ᵃ	❌ attribute ObjectName([Identifier("#0")]) is missing from the schema: cannot resolve attrib	✅	✅
MERGE INTO delta.path“ (path target)	❌ attribute ObjectName([Identifier("#0")]) is missing from the schema: cannot resolve attrib	✅	✅
OPTIMIZE	❌ invalid argument: found OPTIMIZE at 0:8 expected something else, ';', statement, or end of	✅	✅
VACUUM	❌ invalid argument: found VACUUM at 0:6 expected something else, ';', statement, or end of i	✅	✅
Change Data Feed (must not be inert) ᵇ	❌ Table features must be specified, please specify: ChangeDataFeed	✅	✅
readStream (rate source) — schema only ᶜ	✅	✅	✅
format("kafka") + bootstrap/subscribe (rows on the engine) ʲ	❌ No table format found for: kafka	✅	✅
Streaming sink — console — liveness only ᶜ	✅	✅	✅
Streaming sink — memory (rows readable)	❌ No table format found for: memory	✅	✅
Streaming sink — parquet (rows readable)	❌ cannot write streaming data to listing table	✅	✅
Streaming sink — delta (rows readable)	❌ unsupported extension node for streaming: DeltaWriteNode { input: Projection(Projection {	✅	✅
sc / RDD API	❌ [JVM_ATTRIBUTE_NOT_SUPPORTED] Attribute sparkContext is not supported in Spark Connect a	❌ [JVM_ATTRIBUTE_NOT_SUPPORTED] Attribute sparkContext is not supported in Spark Connect a	✅
spark._jvm bridge	❌ [JVM_ATTRIBUTE_NOT_SUPPORTED] Attribute _jvm is not supported in Spark Connect as it dep	❌ [JVM_ATTRIBUTE_NOT_SUPPORTED] Attribute _jvm is not supported in Spark Connect as it dep	✅
createDataFrame(local_rows)	✅	✅	✅
Python UDF	✅	✅	✅
SQL over a temp view	✅	✅	✅
Filter on a row_number() column the SELECT drops ᵈ	✅	✅	✅
CREATE TABLE with no USING defaults to Delta ᵍ	❌ Invalid table location: No commit files found in _delta_log	✅	❌ [NOT_SUPPORTED_COMMAND_WITHOUT_HIVE_SUPPORT] CREATE Hive TABLE (AS SELECT) is not supporte
DESCRIBE TABLE on a registered Delta table ᵉ	❌ DESCRIBE returned 0 rows: []	✅	✅
DESCRIBE DETAIL on a registered Delta table ᶠ	❌ invalid argument: found DETAIL at 9:15 expected 'FUNCTION', 'CATALOG', 'DATABASE', 'SCHEMA	✅	✅
read.text(wholetext=True) — one row per file (must not be inert) ʰ	✅	✅	✅
read.json(multiLine=True) over a JSON-array file ʰ	❌ Json error: Not valid JSON: EOF while parsing a list at line 1 column 1	✅	✅

3 of 26 capabilities differ between the engines. Those are precisely the rows the JVM overlay exists for, and the candidate list for upstream Sail contributions.

ᵃ Qualified deliberately. e2e/sail proves the same MERGE succeeds on Sail when the registered table is backed by an az:// OneLake URL — the path the emulator actually uses. Only the local-path form fails on the bare engine, so the Sail column is not evidence that Sail lacks MERGE.

The middle column intercepts this probe shape: a subquery source and INSERT *, the same split CTAS uses (SELECT on the engine, upsert through delta-rs). Named-source MERGE (the medallion shape) was already intercepted. WHEN MATCHED THEN DELETE still falls through.

ᶜ These two assert less than the others, and say so rather than implying more. Every other streaming row now proves rows actually reached the sink — files read back, or a queryable table — after a green console cell was found to mean only that a query object reported itself active while the engine delivered nothing. No stronger assertion exists from a Spark Connect client for these two: console writes to the server’s stdout, a readStream is unobservable without a sink, and Sail reports no progress metrics (lastProgress is None, recentProgress is empty) — so asserting on those would fail for a missing API rather than a missing capability.

ⁱ The three durable sink rows (memory / parquet / delta) on the middle column wrap writeStream.start(): Sail’s streaming rate plan is collected as a bounded query (limit(n).collect(), measured 2026-08-14) and the rows are batch-written (or registered as a temp view). Announced on stderr. There is no checkpoint; a continuous query lands one micro-batch. Bare Sail still fails with DeltaWriteNode / listing table / No table format found for: memory. foreachBatch on an engine stream falls through. Native format("kafka") (source and sink) is ʲ. Native checkpointed streaming is the JVM overlay.

ᵇ The CDF row on the middle column intercepts both halves of the notebook API: a Delta write with delta.enableChangeDataFeed (and later appends to that table) and a read with readChangeFeed, announced on stderr. The result is materialised (createDataFrame); .explain() is a LocalRelation. Bare Sail still cannot enable the table feature (Unsupported table features required: [ChangeDataFeed]), and e2e/sail — which does not install this interception — still shows the unwrapped readChangeFeed option as accepted and inert. The helper spark.delta_change_feed remains, verified in e2e/livy against OneLake. Native CDF on a classic session is the JVM overlay.

ᵈ This row is green, and it is here to keep it that way — it records a capability we briefly believed was MISSING. The dbt-fabricspark medallion models failed with attribute ObjectName([Identifier("_rn")]) is missing from the schema: cannot resolve attribute and were rewritten into two CTEs, with the cause attributed to Sail. This probe is that exact shape — rank into _rn, keep _rn = 1, project only the real columns — and Sail executes it correctly over Spark Connect. So does a plain unprojected filter, and so does the view-materialised form.

The attribution was therefore wrong, or at least unproven: whatever rejected _rn lives on the Livy path (emulator to spark-agent to Sail) or in dbt’s generated SQL, not in Sail’s SQL support. Not yet localised further — dbt projects an explicit column list over a wide Delta source, which this probe does not reproduce at that width. The two-CTE rewrite is portable and costs nothing, so it should stay; what should not stay is the belief that Sail cannot do this.

ᵉ The real cause of the _rn failure ᵈ was chasing. Sail answers DESCRIBE TABLE on a catalog-registered Delta table with the correct SCHEMA and ZERO ROWS, and raises nothing. Against a temp view it answers correctly, so the gap is specific to registered tables — which is exactly what the emulator’s registerLakehouseTables creates for every lakehouse table, so it reaches any consumer that introspects, not just dbt.

It stayed invisible because every component behaved reasonably. Sail returns no rows and no error; the Livy agent forwards that faithfully (the schema has three fields, so it is not an empty envelope); dbt’s get_columns_in_relation reads it as “this table has no columns”; the model’s Jinja loops over an empty list and emits nothing; the compiled SQL becomes select followed by from. Nobody lied — the information that the answer was MISSING did not survive the chain. A consumer cannot tell “no columns” from “the engine did not answer”, and that indistinguishability is the actual defect.

The medallion models now read columns from run_query("select * from t limit 0").column_names, which carries the schema in the result envelope and never asks the catalog. This row going green on the middle column is the emulator answering from the Delta log once the table’s LOCATION is recorded (registerLakehouseTables, or CREATE TABLE … USING delta LOCATION passing through spark.sql). The Sail column stays the zero-row gap; that is still what a bare engine returns. A name nobody recorded still falls through to Sail.

ᵍ Delta-by-default is a FABRIC property. Bare Sail and the JVM overlay both default to Hive, so a CREATE TABLE … LOCATION with no USING does not write Delta on either engine. The middle column honours an explicit LOCATION and writes Delta — more faithful to Fabric than the JVM overlay, which is why this row can be ❌ / ✅ / ❌.

dbt-fabricspark’s file_format_clause macro emits NO clause for exactly one value of file_format: delta, the one the adapter assumes is the default. So a model configured +file_format: delta with +location_root pointing at the lakehouse emitted create or replace table ... location '...' as ... with no USING, and the lakehouse never received silver while dbt reported success. Two rounds of debugging went into a config that was being applied correctly the whole time: the same value that proved it was applied was the value that suppressed the clause.

The fabricspark__file_format_clause override in the examples is still required on bare OSS Spark, including the JVM overlay. It is not a Sail workaround waiting for a better engine — it is the price of running dbt-fabricspark anywhere that is not Fabric (or this emulator’s LOCATION interception). That is worth knowing before someone deletes it as Sail-specific.

ᶠ The same question asked of the OTHER introspection route. On bare Sail it fails the opposite way from ᵉ: DETAIL is not in the DESCRIBE grammar at all, so this RAISES rather than returning nothing. That is the better failure — a collect()[0] never runs on an empty list; instead it gets a parse error naming the statement.

The middle column answers from the Delta log once the table’s LOCATION is recorded, the same way ᵉ does. Named OPTIMIZE/VACUUM use that cache (remember()), not a round-trip through Sail’s missing grammar. A name nobody recorded still falls back to DESCRIBE DETAIL and fails loudly.

Kept as its own row rather than folded into ᵉ because the contrast is the lesson: the silent one cost a day, the loud one cost minutes, and they are the same missing capability on the bare engine.

ʰ These two arrived as a report that Sail ACCEPTS AND IGNORES both options. Measuring it found one real gap and one false alarm, and the false alarm is the more useful half.

multiLine on the middle column wraps DataFrameReader.json for the named option only, parses the file on the driver (including a Spark text-writer directory: part file in, _SUCCESS skipped), and materialises via createDataFrame. Announced on stderr; .explain() is a LocalRelation. Plain json(), multiLine=False, a list of paths, and a schema fall through. Bare Sail stays NDJSON-only: a file that is one JSON array cannot be parsed at all, while Spark JVM reads it natively. The exact message depends on the fixture’s shape, so read the Sail-column cell as evidence of NDJSON-only parsing rather than as a fixed string — the array here spans lines, so the reader fails on a bare [; an array on ONE line fails differently (Expected JSON record to be an object, found Array). The multi-line fixture is kept because it exercises both halves of the option: spanning lines AND being an array.

wholetext is GREEN on Sail, and this row exists to keep it that way — like ᵈ, it records a capability we were told was missing and is not. The reported symptom (one row per LINE, on both engines) reproduces exactly, but it is a PySpark spelling artifact rather than an engine limit: DataFrameReader.text(path, wholetext=False, ...) passes that DEFAULT into _set_opts, which overwrites any .option("wholetext", ...) set beforehand. So the .option() form cannot take effect for this reader on ANY engine. Measured on one JVM session against one file: .option() -> 3 rows, text(p, wholetext=True) -> 1 row. Both probes therefore use the keyword form.

That is why the JVM column is worth reading first. A probe written the .option() way turns every engine red, including the reference one, and a red reference cell is the tell that the probe is measuring itself. The companion trap is the fixture: with ONE line per file, honoured and ignored give an identical row count, which is how the original report first concluded wholetext WORKED. Both probes therefore use a multi-line fixture and assert the PLAIN read first, so a broken fixture reports itself as a broken fixture rather than as a missing capability — the same discipline as the streaming rows ᶜ, which assert rows reached the sink rather than that a query started.

The fixtures are written through the ENGINE, not with a client-side open(): on the sail profile the probe runs in a different container and shares no volume with the engine, so a client-written file would not be there to read.

Reported from contoso-data-platform, where the multiLine workaround is to read the page as text and parse in-engine (from_json + explode), keeping the data path distributed. That has its own trap: from_json returns NULL on a schema mismatch rather than raising, and the row count comes from the array’s length rather than its contents, so every count-based assertion still passes while every column is empty.

ʲ OSS spark.read.format("kafka") with kafka.bootstrap.servers + subscribe (the measured probe). The middle column consumes on the driver (kafka-python) and createDataFrames the Kafka schema (key/value/topic/partition/offset) into Sail; CAST(value AS STRING) therefore runs on the engine. Announced; one micro-batch; .explain() is a LocalRelation. Not rate. The same wrap honours subscribePattern, assign, JSON startingOffsets/endingOffsets, includeHeaders, SASL PLAIN, GSSAPI, PEM SSL, JKS/P12 truststores, and a kafka sink (write/writeStream.format("kafka")) — unit-tested, not this probe. Bare Sail has no kafka source. JVM uses spark-sql-kafka. Checkpointed streaming stays on the overlay.