Real-Time Intelligence: a real KQL engine behind Eventhouse

Status: shipped, opt-in sidecar, graded Real. Eventhouse / KQL Database execution is real when Microsoft’s own KQL engine container is attached, and an honest 501 when it is not. --profile rti stays opt-in — the engine needs AVX2, which Rosetta does not provide. This is Tier 2, item 2 of 24-parity-completion.md — the same move the repo made for T-SQL (terminate the protocol ourselves, byte-relay to a real SQL Server) and for Spark (drive a real Sail/JVM engine behind the Livy contract).

The shape of the real thing

Fabric does not put KQL on the control plane. An Eventhouse item publishes a cluster URI, and clients then speak the Kusto REST protocol to it — the sequence Microsoft’s own tutorial follows (fabric-docs real-time-intelligence/eventhouse-deploy-with-fabric-api.md, .../map/tutorial-create-real-time-map-python.md):

GET  /v1/workspaces/{ws}/eventhouses/{id}
     → properties.queryServiceUri, properties.ingestionServiceUri,
       properties.databasesItemIds

POST {queryServiceUri}/v1/rest/mgmt    {"db": "<database displayName>", "csl": ".create-merge table T(…)"}
POST {queryServiceUri}/v1/rest/query   {"db": "<database displayName>", "csl": "T | count"}
POST {queryServiceUri}/v2/rest/query   (the frame-stream dialect the SDKs use)

Creating an eventhouse also creates a default child KQL database with the same name (create-eventhouse.md), and further databases are created with a creationPayload carrying parentEventhouseItemId. All of that is implemented (internal/api/kql.go); the KQL itself is not.

The engine

mcr.microsoft.com/azuredataexplorer/kustainer-linux:latest — Microsoft’s own Kusto engine in a container (the Azure Data Explorer “Kusto emulator”).

Start	ACCEPT_EULA=Y is mandatory; the image exits 13 without it
Port	8080, HTTP only — the engine supports neither TLS nor Entra
Auth	none; it is a trusted sidecar on the compose network
Readiness	POST /v1/rest/mgmt {"csl":".show cluster"} → 200 (needs no database)
Ingestion	no data-management service: queued ingestion and the ingest- endpoint are unsupported. Direct ingestion commands (.set-or-append, .ingest inline) run on the engine itself
Memory	4 GB recommended (Microsoft’s own examples pass -m 4G)
Platform	linux/amd64, and it needs AVX2 — the constraint is the instruction set, not the host. The engine’s native layer aborts without AVX2 (libKusto.NativeInfra.so failed … Crash_FailSlow). Rosetta translates x86-64 but stops at SSE4.2, so --platform linux/amd64 on Apple silicon crashes on boot; a QEMU x86-64 VM with --cpu-type max supplies AVX2 and the engine runs. Both directions verified on this repo’s own hardware — see below.

What the emulator does (and does not)

The emulator terminates Fabric’s half of the contract and relays the KQL:

• Bearer validation on the Kusto audience (https://kusto.fabric.microsoft.com, plus ADX’s https://api.kusto.windows.net) — a control-plane token is rejected, as in real Fabric.
• Workspace RBAC: a management command mutates and needs Contributor; a query needs Viewer.
• Isolation: each Fabric KQL Database gets its own engine-side database, named from the item id, created on first use. Two databases under one eventhouse cannot see each other’s tables even with identical names. The internal name is mapped back to the Fabric display name on the way out, so a client never sees it.
• Honest failure: no engine → 501 in Kusto’s own error envelope; an unreachable or erroring engine → 502. Never a synthesized result set.

Everything past that — parsing, planning, evaluating, ingesting — is the engine’s. The emulator has no KQL implementation and will not grow one.

Endpoint addressing

Real Fabric gives every eventhouse a host of its own (<guid>.z<n>.kusto.fabric.microsoft.com). A local emulator has no wildcard DNS, so the cluster URI is path-prefixed on the emulator’s own origin:

{scheme}://{host}/kusto/{workspaceId}/{eventhouseId}

This is transparent to real clients: every Kusto SDK builds its endpoints by appending /v1/rest/… or /v2/rest/query to the cluster URI. Microsoft’s azure-kusto-data drives this URI unmodified in e2e/rti. It is the same accommodation OneLake makes with its account-prefixed path form.

Running it

docker compose --profile rti \
  -f docker-compose.yml -f docker-compose.override.yml -f docker-compose.rti.yml up

Two opt-ins, doing different jobs: --profile rti starts (and only then pulls) the ~1.2 GB engine, and -f docker-compose.rti.yml hands the emulator its URL (FABRIC_KQL_URL, or --kql-url). Without the profile a plain docker compose up never touches the image; without the overlay the Kusto routes 501.

The witness

e2e/rti (CI job rti) is the oracle, and it asserts real execution, not plumbing: create a table, ingest rows two documented ways, then query them back and check the values — a summarize avg(…) by …, a datetime filter, and a cross-database isolation check. Two independent client families run over the same surface: raw REST, and Microsoft’s own azure-kusto-data SDK (which parses the v2 frame stream and so exercises a second dialect).

It also settles a question no fake engine can: which column names KQL actually refuses. Some KQL keywords cannot name a column in a schema declaration — the engine answers SYN0002 — and the Eventstream → Eventhouse drain builds its .create-merge table T (name:type, …) from whatever fields the events carry, so kind is an ordinary field name that would emit a command real Kusto rejects. The emitter quotes every name (['kind'], Kusto’s documented remedy), and the witness probes 36 candidate keywords against kustainer in both directions.

The answer is narrower than the documentation implies. Microsoft’s identifier-naming rules say a keyword used as an identifier must be quoted, but this engine refuses only nine bare — and, between, false, kind, or, order, project, true, union — and takes the other 27, where and summarize among them. All 36 are legal quoted. So the witness asserts set equality: a keyword refused that was not, or accepted that was not, fails the job by name. internal/api/kql_test.go’s kqlKeywords is that set and nothing more, which keeps the in-process stand-in from refusing KQL the engine runs — the failure mode that matters more here than the one it guards against.

The destination table name is the same question in a different position in the grammar, and the drain takes it from the bind against the same character class. The witness settles that one too: bare kind refused, ['kind'] accepted in both .create-merge table and .ingest inline into table, and ['Readings'] shown to merge into the table this suite created bare — three columns and four rows, unchanged. That last one is what licenses quoting every drain’s table name rather than only the ones that look dangerous.

That CI job is the witness of record: it runs on amd64 runners, and the witness depends on kustainer: service_healthy, so an engine that never comes up fails the job rather than passing quietly.

Running it on Apple silicon

The default Docker setup on an M-series Mac translates amd64 with Rosetta, which stops at SSE4.2 — so the engine crashes during boot. e2e/rti/run.py detects this and explains it, rather than letting you discover it as a native crash mid-build. Note it inspects the Docker daemon’s architecture, not the host’s: with the VM below the two deliberately differ.

QEMU does implement AVX2, so a real x86-64 VM works. --cpu-type max is the part that matters — QEMU’s default CPU model omits AVX2 as well:

brew install colima qemu lima-additional-guestagents
colima start --profile fabric-x86 --arch x86_64 --vm-type qemu \
  --cpu-type max --memory 20 --cpus 6 --disk 60
export DOCKER_CONTEXT=colima-fabric-x86
python3 e2e/rti/run.py

lima-additional-guestagents is not optional: Lima 2.x ships only the host architecture’s guest agent, and without it the VM fails to start with guest agent binary could not be found for Linux-x86_64.

Size the VM generously: at 8 GiB the engine is OOM-killed (exit 137) once the rest of the stack is running.

Measured on an M4 Max (36 GB): the VM exposes avx2, the engine is ready in ~40 s, and the full suite passes with the same values CI asserts. The slow parts are the ~3.4 GB image pull and the image builds — everything is translated — not the engine; budget ~10 min for a cold run and far less once cached. CI remains the witness of record.

Boundaries (deliberate, not backlog)

• Eventhouse streaming ingest / queued Kusto.Ingest stays 🔴. The engine is a query/ingest engine; Fabric’s streaming-ingest protocol and the ingest- endpoint are a different service, and the Kusto emulator explicitly has no streaming ingestion. Eventstream → Eventhouse destination is a separate green row: Custom HTTP produce drains through direct ingest (.create-merge + .ingest inline) against the already attached engine — the same path this sidecar already supports. That is produce-triggered, not a streaming pipeline. 51-eventstream-kafka.md.
• Queued ingestion / the ingest- endpoint and Kusto.Ingest SDKs: unsupported by the engine. ingestionServiceUri therefore points at the engine, and direct ingestion commands are the supported path.
• OneLake availability (a KQL database mirroring itself into OneLake Delta) is not wired: the engine writes to its own storage, not ours.