Skip to content

How Pipemason verifies results

The runner measures. A run cannot reach done on its own say-so; it has to pass three checks the runner performs itself.

Before a run is reported complete, the runner re-runs the configured commands, checks that new tests actually executed, and validates a verdict for every acceptance criterion. What it measures is what the dashboard, the export, and pipemason status show.

1. Verified finalize

When a run reaches finalize, the runner executes each active domain’s commands.<domain>.typecheck and commands.<domain>.test from .pipeline/config.yml in the run’s working tree, with a bounded timeout. It records the exit code, duration and, for recognised runners (vitest, jest, playwright, pytest, go test), the parsed pass and fail counts.

  • Results are written to the verification: block in .pipeline/STATE.md and sent to the cloud as one verification_result event per domain.
  • Any non-zero exit fails the run’s verification. The terminal and JSON summaries print the measured numbers, and pipemason resume continues from verify once you have fixed the cause.
  • A domain with no test command is recorded as skipped and shown as such. It is never treated as passing.

2. Orphan tests

A test that never runs is worse than no test: it looks like coverage. The runner lists test files added or changed in the run (*.test.*, *.spec.*, tests/**, __tests__/**, test_*.py, *_test.go) and checks whether the domain’s configured test command executes each one. Files that do not run, for example vitest specs in a package whose test command is playwright, are reported in an orphan_tests event and in the terminal.

Under the default policy verification.orphan_tests: fail, orphan tests fail finalize verification. Set it to warn to report without blocking. Pipemason writes tests only where the configured command runs them, and escalates when the framework is missing rather than invent one.

3. Acceptance-criteria scorecard

The acceptance criteria in .pipeline/spec.md (AC-1 to AC-n) are first-class data. At verify and again at finalize, every criterion is given one verdict: met, unmet, or untested, with evidence required for met. The runner checks that every criterion has a valid verdict; if it does not, the run escalates.

  • Each verdict is sent to the cloud as an acceptance_result event and stored per run, so the run page shows the scorecard as its headline block and the program page rolls it up per story. The run export carries an acceptance section.
  • Any unmet criterion blocks done. untested is allowed but always visible.
  • pipemason status prints the verification block and the scorecard.

Note

Verification adds one full test run per active domain at the end of the run. Raise verification.command_timeout_ms if a suite legitimately takes longer than 20 minutes.

Drift checks

At the exit of each of the tdd, implement, e2e, and ui phases, before the run advances to the next phase, Pipemason runs one drift check. It is a checkpoint, not continuous: it runs once when a phase finishes, not in the background on every edit.

The check compares the changes the phase made against the acceptance criteria in .pipeline/spec.md and the interfaces agreed in the contracts phase.

Drift is exactly one of four conditions:

  • ac_not_advanced — the changes fail to move the phase’s named acceptance criteria forward.
  • unrequested_scope — the changes implement something the spec did not request.
  • unreferenced_module — a module or file the phase added is never imported, called, or registered.
  • contract_contradiction — the changes contradict the interfaces agreed in the contracts phase.

Each check ends with one of three verdicts:

VerdictWhat happens
on_trackThe run advances to the next phase.
driftThe phase is sent back with the findings to correct them, up to max_bounces times (default 2). If drift remains after that, the run escalates to you.
blockedThe check could not complete. The result is logged and the run advances. Never a stop.
  • Results are visible in the drift: block of .pipeline/STATE.md, in one drift_check event per check in iterations.log, on the dashboard live feed, and in the run page’s “Drift Check” block.

Note

Drift checks are an earlier, softer signal, not a fourth done-gate: the finalize gate above is unchanged and remains the hard stop.

Configuration, with its defaults:

drift_check:
  enabled: true                       # drift_check.enabled -- master switch
  phases: [tdd, implement, e2e, ui]   # drift_check.phases -- subset of these four
  max_bounces: 2                      # drift_check.max_bounces -- bounces before escalating

See .pipeline/config.yml for the full reference.

Configuration

verification:
  command_timeout_ms: 1200000   # per command, default 20 minutes
  orphan_tests: fail            # fail | warn

See .pipeline/config.yml for the full reference.