How Pipemason verifies results
The runner measures. A run cannot reach done on its own say-so; it has to pass three checks the runner performs itself.
Before a run is reported complete, the runner re-runs the configured commands, checks that new tests actually executed, and validates a verdict for every acceptance criterion. What it measures is what the dashboard, the export, and pipemason status show.
1. Verified finalize
When a run reaches finalize, the runner executes each active domain’s commands.<domain>.typecheck and commands.<domain>.test from .pipeline/config.yml in the run’s working tree, with a bounded timeout. It records the exit code, duration and, for recognised runners (vitest, jest, playwright, pytest, go test), the parsed pass and fail counts.
- Results are written to the
verification:block in.pipeline/STATE.mdand sent to the cloud as oneverification_resultevent per domain. - Any non-zero exit fails the run’s verification. The terminal and JSON summaries print the measured numbers, and
pipemason resumecontinues fromverifyonce you have fixed the cause. - A domain with no test command is recorded as
skippedand shown as such. It is never treated as passing.
2. Orphan tests
A test that never runs is worse than no test: it looks like coverage. The runner lists test files added or changed in the run (*.test.*, *.spec.*, tests/**, __tests__/**, test_*.py, *_test.go) and checks whether the domain’s configured test command executes each one. Files that do not run, for example vitest specs in a package whose test command is playwright, are reported in an orphan_tests event and in the terminal.
Under the default policy verification.orphan_tests: fail, orphan tests fail finalize verification. Set it to warn to report without blocking. Pipemason writes tests only where the configured command runs them, and escalates when the framework is missing rather than invent one.
3. Acceptance-criteria scorecard
The acceptance criteria in .pipeline/spec.md (AC-1 to AC-n) are first-class data. At verify and again at finalize, every criterion is given one verdict: met, unmet, or untested, with evidence required for met. The runner checks that every criterion has a valid verdict; if it does not, the run escalates.
- Each verdict is sent to the cloud as an
acceptance_resultevent and stored per run, so the run page shows the scorecard as its headline block and the program page rolls it up per story. The run export carries anacceptancesection. - Any
unmetcriterion blocksdone.untestedis allowed but always visible. pipemason statusprints the verification block and the scorecard.
Note
verification.command_timeout_ms if a suite legitimately takes longer than 20 minutes.Drift checks
At the exit of each of the tdd, implement, e2e, and ui phases, before the run advances to the next phase, Pipemason runs one drift check. It is a checkpoint, not continuous: it runs once when a phase finishes, not in the background on every edit.
The check compares the changes the phase made against the acceptance criteria in .pipeline/spec.md and the interfaces agreed in the contracts phase.
Drift is exactly one of four conditions:
ac_not_advanced— the changes fail to move the phase’s named acceptance criteria forward.unrequested_scope— the changes implement something the spec did not request.unreferenced_module— a module or file the phase added is never imported, called, or registered.contract_contradiction— the changes contradict the interfaces agreed in thecontractsphase.
Each check ends with one of three verdicts:
| Verdict | What happens |
|---|---|
| on_track | The run advances to the next phase. |
| drift | The phase is sent back with the findings to correct them, up to max_bounces times (default 2). If drift remains after that, the run escalates to you. |
| blocked | The check could not complete. The result is logged and the run advances. Never a stop. |
- Results are visible in the
drift:block of.pipeline/STATE.md, in onedrift_checkevent per check initerations.log, on the dashboard live feed, and in the run page’s “Drift Check” block.
Note
Configuration, with its defaults:
drift_check: enabled: true # drift_check.enabled -- master switch phases: [tdd, implement, e2e, ui] # drift_check.phases -- subset of these four max_bounces: 2 # drift_check.max_bounces -- bounces before escalating
See .pipeline/config.yml for the full reference.
Configuration
verification: command_timeout_ms: 1200000 # per command, default 20 minutes orphan_tests: fail # fail | warn
See .pipeline/config.yml for the full reference.