Continuous integration

make lint          # ruff, ruff format, terraform, ty, annotations, golangci-lint
make test          # unit tests + e2e witnesses
make stack         # everything from nothing: up, seed, apply, verify
make secrets       # gitleaks, over the working tree AND the history
make vulns         # govulncheck, over the Go module

Five workflows run in this repository. Four of them exist because a check nobody runs finds nothing.

Workflow	Trigger	What it is for
ci.yml	push to main, pull request — not prose-only commits	the four jobs below
security.yml	push, PR, weekly	gitleaks and govulncheck. Weekly because vulnerabilities are published against code that has not changed
codeql.yml	push, PR, weekly — not prose-only commits	Python, Go, TypeScript and the workflows themselves
docs-site.yml	changes under docs/, site/, website/	builds and publishes the documentation site
release.yml	a v* tag	both executor images to GHCR, and the GitHub Release — see Releases

.github/workflows/ci.yml runs four jobs on push to main and on every pull request. They are separate jobs rather than one, so a formatting mistake is reported in about a minute instead of after a fifteen-minute bring-up.

Job	Runs	Why it is its own job
quality	ruff, ty, pytest, check-discipline.sh, check_prod_paths --strict, preflight --offline	fast, needs no stack, catches most mistakes
go	go vet, go test, golangci-lint	different toolchain, no reason to wait on Python
infra	terraform fmt -check, terraform validate	proves the production definition still parses and type-checks
stack	make stack, make conformance	the only job that proves the thing actually works

The first three are cheap. The fourth is the one that matters.

What a documentation change costs

A one-word fix to a document used to run every workflow. Measured over recent successful runs:

Workflow	Per run	On a prose-only commit
ci.yml	~1060 s	skipped
codeql.yml	~97 s	skipped
security.yml	~26 s	runs
docs-site.yml	~61 s	runs — the actual publish

So 21 minutes of CI to deliver 61 seconds of work. ci.yml and codeql.yml now carry paths-ignore for docs/*.md, docs/**/*.md, README.md, SECURITY.md, site/** and website/**.

security.yml is deliberately not filtered. .gitleaks.toml explains why in its own words: documents and the README are not allowlisted, because a credential pasted into a document by accident is the most likely way one would ever land here. Twenty-six seconds is not worth that hole.

Three properties hold this up, and the third is the one that had to be fixed before the rest was safe:

• paths-ignore skips only when EVERY changed file matches. A commit touching a document and a source file runs in full. There is no way to smuggle a code change past CI by attaching a doc to it.
• A path not listed runs the workflow. Adding a source directory cannot silently opt out; only the six prose patterns above can cause a skip. The failure direction is “runs unnecessarily”, never “skipped wrongly”.
• docs/*.json is absent from the list on purpose. witnesses.json and coverage.json live under docs/ but are manifests the guards read, and a wrong number in one of them is exactly what the suite exists to catch. A commit touching them runs everything.

The guards did not lose coverage; they gained a second home

check_counts reads README.md, site/index.html and docs/*.md — precisely the paths now filtered out of ci.yml — and it ran nowhere else. Filtering without moving it would have left every prose total unguarded, which is the gate that reports on less than it claims in its most direct form.

So check_counts, check_docs_nav and check_version now run in both workflows, and the two trigger sets are complementary: a prose commit runs them in docs-site.yml, a commit touching a guard script or any source runs them in ci.yml. README.md joined docs-site.yml‘s triggers so that workflow’s trigger set and check_counts’ target set are the same set.

All nine commit shapes were simulated against the globs before the change was pushed, rather than trusted: prose-only skips CI and CodeQL and publishes; doc plus source runs everything; a witness manifest runs everything; a guard script runs CI.

What the scanners have actually found

A scanner is worth its runtime only if it has caught something. These have.

• govulncheck, first run: a reachable vulnerability in golang.org/x/text below v0.39.0, called from both backends through sql.Rows.Columns and sql.OpenDB. Then a better one — CI reported nineteen standard-library vulnerabilities while the same command locally reported none. go.mod said go 1.26.0 and setup-go installs exactly that, while the container had a patched 1.26.x. The shipped binary was never affected, because the Dockerfile uses golang:1.26-alpine, but CI was testing a Go we do not release — which is how a real finding gets dismissed as noise. See The Go toolchain is pinned twice.
• CodeQL, first run: "login.microsoftonline.com" in issuer is true of login.microsoftonline.com.example.net, and the suffix test that replaced it is true of notlogin.microsoftonline.com. Either would have sent a Graph call, carrying a token, to a host we did not mean. Low exploitability — the issuer is operator configuration — but the check read as if it identified a host and did not.
• gitleaks: the history is clean across every commit, with the default rules armed. Its allowlist is six generated paths, all gitignored, and nothing else: docs, README, .env.example and the seeds stay scanned, because a credential pasted into a document by accident is the likeliest way one would ever arrive.

The check that ty cannot make

make lint runs scripts/check_annotations.py, which is not a style rule.

ty reports a signature that contradicts its own body only where a caller violates it. Given a parameter annotated list[list] whose first line is if rows is None, it reports nothing at all — there is no call site to blame. So the contradiction sits in the tree until somebody writes the obvious test, and then CI blames the test rather than the annotation. That is exactly how rows_match and rows_contain failed: the guard had been there since the day each was written, and the test that asserted it was correct.

The check reads the definition instead of waiting for a caller.

Why make stack runs from nothing

A working copy that has been used for a while is not the system a new clone gets. Ours had accumulated state that made it work for reasons no longer written down anywhere, and the first run from an empty state found two breaks that neither code review nor a local make test could see:

• .env.example shipped DAS_APIM_VALIDATE_JWT=true, so a fresh clone told the gateway to validate tokens the pinned emulator cannot validate. Every call came back “Unauthorized. Sign in and retry.”
• The SQL resource app never exposed user_impersonation. Registering a resource makes <resource>/.default issuable but not the delegated scope, so on-behalf-of failed with AADSTS70011 against an empty tenant. Ours had accumulated the scope over dozens of runs.

Both had been true for a long time and were invisible from either working copy. That is the whole argument for the job: the only honest test of a setup procedure is running it on a machine that has never run it.

A third one has the same shape and is worth naming separately, because it is structural rather than accidental. Two sessions sharing one checkout had a ruff format reflow sitting unstaged in the tree. make lint was green for both of them, against a committed state that was red — the working trees agreed with each other, and neither agreed with the repository. Nothing inside the checkout can catch that, because everything inside the checkout is reading the same unstaged file. CI is the only participant that reads what was actually committed.

Keeping the seed idempotent

Because stack starts from nothing every time, anything the seed discovers or creates has to be written down rather than remembered. seed/common.write_env persists created ids into .env — never .env.prod, which is a template and must not acquire values. The harness container waits on service_healthy rather than on container start: docker compose up -d returns before OpenMetadata answers, which is invisible when a person seeds by hand a minute later and fatal every time in CI.

The Go toolchain is pinned twice, and both must agree

services/warehouse-query-go/go.mod names the version, and every CI job reads it through go-version-file rather than repeating it. .go-version pins the same version for goenv so a host build uses what CI uses.

They are separate files because they are read by different tools, which means they can disagree — and the way that surfaces is a security gate rather than a build error. govulncheck reported 19 standard-library vulnerabilities while every build was green: the containers were already on a patched Go, and CI was installing the exact version go.mod asked for, which was not.

Bumping the directive fixed all nineteen. If you see them come back, check whether the two pins have drifted before looking at the code.

goenv install "$(cat .go-version)"

Linting Go locally lints a different thing than CI does

.golangci.yml lives at the repository root, and golangci-lint finds it by walking up from the directory it runs in. Lint the Go executor like this and it never sees the config:

cd services/warehouse-query-go
docker run --rm -v "$(pwd):/src" -w /src golangci/golangci-lint:v2.13.1 golangci-lint run ./...

Only services/warehouse-query-go is inside the container, so the walk finds nothing and the run uses golangci-lint’s defaults. It reports 0 issues while CI reports real ones — the pinned version is the same, the linter set is not. Mount the root instead:

docker run --rm -v "$(pwd):/src" -w /src/services/warehouse-query-go \
  golangci/golangci-lint:v2.13.1 golangci-lint run ./...

That is the same class of trap as the Go toolchain being pinned twice: local and CI disagree, local is the more convenient one to believe, and the disagreement is silent in the direction that lets you push. It cost a red build on a nilerr finding that a correctly-mounted run would have shown in three seconds.

Building the docs site locally

make docs             # pnpm on the host — fast, for writing
make docs-container   # Docker and nothing else — what CI does

Two entry points, because the obvious single setup fails either way.

The host build exists because the site is static and a container round-trip per edit would make writing docs slower than writing code. The container build exists because Docker and nothing else is what the rest of this repository promises, and a fresh clone should not need a working Node install.

The trap is that node_modules holds platform-specific binaries, and a bind-mounted checkout hands host and container the same directory. Whoever installs last breaks the other, and the error names esbuild:

You installed esbuild for another platform than the one you’re currently using.

which reads as a broken dependency rather than a bad install — and the lockfile is innocent, because it records every platform. This has now cost a day in each direction: a container install broke the host build, and later a host install broke the container build.

So the container gets its own node_modules, in named volumes mounted over the bind mount. Both builds work, neither can overwrite the other’s binaries, and the volumes persist so the install is paid for once. make docs-clean drops them.

The general shape is worth keeping in mind beyond this repository: a bind mount shares a directory, and any directory holding compiled artefacts is shared state, not shared source. Mounting a volume over it is how the two stop fighting.

rm -rf node_modules website/node_modules && pnpm install --frozen-lockfile

While it is broken, the docs build is CI-only — so an unlinked chapter or a stale number on the front page is not caught until after a push. Both of those checks exist because both have happened.

A red job is not always where the cause is

The quality witnesses re-run ruff and ty inside the stack job. So a single unformatted line fails two jobs: Lint, types and guards in about a minute, and Full stack witnesses twelve minutes later. They look like two problems and are one.

That is only half of it. When the stack job fails, the workflow dumps container logs — and those logs contain the executor’s audit lines, including refusals that witnesses deliberately provoke. services/conformance/run.py mints a token for a persona with no role on the source and asserts it is turned away, so a passing run prints:

"verdict":"denied", "reason":"… the principal has no role on the workspace
of \"contoso_warehouse\""

Read next to a failing job, that looks like the cause. It is the security property working. This has already produced a wrong diagnosis that reached three sessions: the stack job was red on a ruff format miss, and the log ended in deliberate refusals.

So: read the FAIL lines, not the log tail. Every witness failure prints FAIL [phase] name — detail. If there is no FAIL line in the stack job, the job failed on a gate — formatting, types, the manifest — not on a witness. And before believing a theory about a recurring failure, check it against each commit: the refusal message above appeared in exactly one of the nine red builds it was blamed for.

A gate that reports on less than it claims

Three of these landed in one day, in three parts of the repo, found by three different people. That is a pattern rather than three bugs, and it is worth recognising by shape because none of them was red:

• The conformance runner’s --expect-executor read docker compose ps and an image label from inside the tools container, which has neither the docker CLI nor the socket. The call raised, the exception was swallowed, and the answer became “the stack is running something unrecognised” — the stack blamed for the harness being unable to see. It had never returned a verdict on any machine, and the parity doc called the phase done.
• The quality witness ran three of make lint’s twelve stages while its own docstring said it ran “the same commands make lint runs”.
• check_counts listed a page whose markup its pattern could not match, so the page was counted as checked and never read.

The common shape: the failure mode is silence, not a failure. A check that cannot run looks exactly like a check that passed, and a check covering three of twelve looks exactly like one covering twelve. Nothing goes red, so nothing prompts the look that would find it — and the surrounding documentation, written from intent, actively discourages that look.

Two habits that catch it, both cheap:

1. Watch a new gate fail on purpose, once. Not “the suite is green” — arrange the condition it exists to catch and see it go red, by name. Every one of the three above would have died in its first minute.
2. Prefer deriving a list to restating one. A gate list written down in a second place drifts on whoever edits the first place next, and they will not know the second place exists. e2e/run.py now derives its stage list from the lint: recipe and fails on a line it cannot classify, so widening make lint cannot silently widen the gap.

And when documentation states coverage, it must enumerate what is not covered. “Runs the same commands make lint runs” is the sentence that kept anyone from checking for a year’s worth of drift in an afternoon.

Two of these are different species, and the fix differs

The three above read alike and are not. Separating them matters because the second kind is the more dangerous and the cheaper to prevent.

• The check never ran. check_counts had been a stage of make lint for two hours before someone asserted it was in no local command — an assertion about the repository that one command would have settled. The fix is behavioural: run the thing before claiming what it does.
• The check ran somewhere it could not see what it was checking, and reported what it could not see as the failure. --expect-executor did this. It is worse than silence, because silence prompts nothing while a confident wrong cause sends the next person somewhere specific and wrong — “the stack is running something unrecognised” is a sentence you act on.

The second has a structural fix, and it generalises past this repo:

A check must be able to observe what it asserts, and must fail differently when it cannot.

“Unrecognised” collapsed the stack is wrong and I cannot see the stack into one verdict. Those have different causes, different fixes, and different people to tell. The same split is why e2e/run.py reports an unclassifiable recipe line separately from a stage that is neither run nor excused: one means the parser is behind the Makefile, the other means a gate is unaccounted for, and a single combined message would get diagnosed as whichever the reader saw last.

A fourth, found the same week: one red gate hiding eight

ty failed at step 6 of the quality job’s fourteen, and steps 7 through 14 were skipped — check_annotations, check_counts, check_docs_nav, check_version, the coverage floor, check-discipline, check_prod_paths and the production-settings check. None of them ran, and none of them said so.

This is the section’s own shape wearing CI’s clothes: in a job summary a skipped step and a passing step are both “not the problem”. The eight gates are independent of each other, so sequencing them behind one ty invocation bought nothing and cost the ability to see seven other results. They run independently now, in both the quality and Go jobs, each guarded only on setup — a failed uv sync should produce one cause, not eight copies of it.

The rule that falls out: steps that do not depend on each other must not be able to mask each other. If a job’s steps are a straight line, one failure tells you about one gate, and the rest is unmeasured rather than green.

The working tree is not the repository

A sibling of the shape above, and the more frequent one here: a claim derived from files on a disk rather than from what is committed. It has cost this repository four times, in three different forms.

• Witnesses. Two phase-15 witnesses and one phase-16 witness asserted on catalog and file state a manual run had left behind. Each passed on the author’s machine and failed in CI, which starts clean every time. The rule that came out of it: a witness must create whatever it asserts on, in the same function.
• The build. A sidebar entry was committed without its page. Astro reads the filesystem, the author had the file, CI did not — and the docs site went down. scripts/check_docs_nav.py exists for this, and judges git ls-files rather than the directory for exactly that reason.
• In prose. A session read --assert-executor in a dirty working tree, concluded it was a pre-existing design that someone had failed to wire up, and said so to its author. It had been written minutes earlier, in the commit under discussion. The account was more damning than the truth and would have become received history had the author not checked git log -S and pushed back.

That last one is the reason this section is here rather than in a commit message. The first two cost a red build, which announces itself. A wrong account of why something broke does not: it gets repeated, and it lands in a document like this one.

The habit is one question, asked before describing what the repository does or has done — what does git say?

git ls-files <path>              # does it exist for anyone but me?
git log -S'<string>' -- <path>   # when did this actually appear?
git show <sha>:<path>            # what did it look like then?

git status reporting a file as modified is a fact about a laptop. It is not a fact about the project, and on a shared checkout it is frequently a fact about somebody else.

Committing when several sessions share the checkout

git add <path> is not the safe operation it looks like. A path names a file, and a file can hold another session’s uncommitted work — that is how a 99-line plan section landed under a commit message about a witness manifest.

Two habits, in order of reliability:

git commit -m "…" -- path/one.py path/two.md   # ignores the index entirely
git diff --cached --stat                       # before every commit, without exception

The pathspec form has no staging step to race, so a concurrent git add from another session cannot be swept in; anything already staged stays staged. It does not work for a file git does not yet track — git commit -- new.py errors with pathspec did not match any file(s) known to git, so a new file still needs git add, and still needs the diff check.

The diff check is the one that always works. Treat an unexpected file, or an unexpected line count, as a stop signal rather than a curiosity.

A pathspec protects other files, not other changes in the same file

This is the limit of the advice above, and both of the sessions recommending it to each other walked into it the same afternoon. git commit -- main.go takes everything in main.go, including the 73 lines of somebody else’s credential work sitting in it. Inside one file the pathspec form is exactly as unsafe as git add; a directory pathspec is worse again, because it silently widens to every contended file underneath.

Two ways out, both of which commit only your hunk and leave the working tree exactly as you found it:

git show HEAD:path/file.py > /tmp/head       # the file as committed
# apply ONLY your edit to that copy, then:
diff -u /tmp/head /tmp/head_fixed > mine.patch
git apply --cached mine.patch                # index = HEAD + your hunk
git commit -F msg                            # worktree untouched

# Or, when several files are involved, build the commit in a private index:
export GIT_INDEX_FILE=/tmp/mine.index && git read-tree HEAD
blob=$(git hash-object -w /tmp/your_version_of_file)
git update-index --cacheinfo 100644,"$blob",path/file.go
git commit-tree $(git write-tree) -p HEAD    # then update-ref

The second leaves the SHARED index stale against the new HEAD, which shows peers a staged revert of your own commit. git reset -q -- <paths> after pushing puts it right without touching anyone’s working tree.

Then verify the COMMIT rather than the working tree, because the working tree contains other people’s changes and will happily build when the commit does not:

git worktree add --detach /tmp/verify HEAD && cd /tmp/verify && <build and test>

Commit messages

Subjects under 72 characters, imperative or descriptive, no body unless the subject cannot carry why the change was made. Rewriting a message after the fact is git commit-tree plus git update-ref, not filter-branch: it preserves author and committer dates exactly and never touches the index or the working tree, so it is safe to do while work is in progress.