Testing, and what counts as evidence
make test # go test ./...make witnesses # every 🟢 parity claim names a CI job that proves itThe parity map is measured, not written
Section titled “The parity map is measured, not written”docs/parity.md is generated by e2e/parity/run.py, which runs every probe
against the built image and writes the table from the answers. --check
regenerates it and fails if the committed file disagrees — that CI job is what
makes the map a claim rather than an aspiration.
Every probe is a statement real Snowflake accepts. A probe is not “something we support”; it is “something a consumer may legitimately write”, and the emulator either answers it or is recorded as not answering it. Four probes invert that and assert a refusal — unparseable SQL, an external stage, an unsupported format option — because answering those would be the defect.
It runs against the image, and that is load-bearing
Section titled “It runs against the image, and that is load-bearing”The duckdb CLI the Dockerfile pins exits 0 after refusing a statement; a newer one on a developer’s machine exits 1. Probing a host build reports honest failures the shipped image does not give. The parity CI job therefore installs no duckdb of its own, and a comment in the workflow says why.
TestEveryDuckdbPinIsTheSameVersion fails if the Dockerfile and the CI steps
ever name different versions, so an emulator cannot be built on one and tested
against another.
Every green claim names a witness
Section titled “Every green claim names a witness”docs/witnesses.json maps each 🟢 to a CI job or a Go test, and
scripts/check_witnesses.py fails if a claim has no witness, if a witness
names a job that does not exist, or if a witness exists for a claim that does
not. It is generated from the same probe run as the parity map.
The e2e suites
Section titled “The e2e suites”| suite | what it proves |
|---|---|
e2e/sdk | the official connectors log in and query |
e2e/sql | warehouse handle, dialect: duckdb, COPY INTO |
e2e/dbt | unmodified dbt-snowflake, confirmed by a separate duckdb binary |
e2e/iceberg | CREATE ICEBERG TABLE is listed by a Polaris catalog |
e2e/target | the `emulator |
e2e/parity | the whole surface, against the image |
The rule they share: the engine that wrote is never the one that confirms.
What runs when
Section titled “What runs when”A change that touches only prose does not run the Go tests, the e2e suite,
CodeQL or the linters. It runs the docs site build, which is the check that
bears on it, and merging it to main publishes the site.
The filter is written as '**' followed by the prose carved out of it, rather
than a list of code paths. The direction is deliberate: a file nobody
anticipated runs the full suite. Running CI on a README costs a minute; not
running it on a source file costs a release.
docs/parity.md and docs/witnesses.json are exempt from the exemption.
They are generated by e2e/parity/run.py, and the parity job exists to fail
when they disagree with what the emulator answers. Treating them as prose
would let a hand-edited parity table merge with nothing run against it — this
project’s central claim about itself, quietly untrue. They are re-included
after the !docs/** exclusion, and tests/test_workflow_paths.py fails if
that ever stops being so.
That test also asserts all four filtered workflows carry the same list. Actions will not resolve a YAML anchor across files, so the list is copied four times, and copies drift.