The measurement harness

The harness is Gettier's measurement face: it replays a frozen scenario against one or more models, in paired arms, and scores the outcome against a four-way taxonomy. It shares the core engine with the proxy — the gated arm runs the very same turn state machine — so a result here is a statement about the product, not a lookalike.

pnpm --filter @gettier/harness start -- \
  --scenario fixtures/gtr-001 \
  --models claude-opus-4-8,gpt-5.6-sol \
  --arms raw,gated,static \
  --runs 5

Each (model × arm × run) combination brings the fixture up, restores it to its frozen state, drives the model, applies any proposed fix, runs the acceptance test, and scores it.

Flags

Flag	Default	Meaning
--scenario	fixtures/gtr-001	Scenario/fixture directory (scenario.yaml + sensors.local.yaml).
--models	claude-opus-4-8	Comma-separated model ids. Each is checked reachable before any call is spent.
--arms	raw,gated	Any of raw, gated, static (see below).
--runs	1	Repetitions per cell — the basis for variance (see Multi-run variance).
--data-dir	./data	Where the run read-model ledger lives (also GETTIER_DATA_DIR).
--tenant	local	Tenant the runs are stored under (also GETTIER_TENANT).

Upstream keys come from the environment or a repo-local .env (ANTHROPIC_API_KEY / OPENAI_API_KEY); values are never logged. Model → provider routing is covered in Setup → Models & provider routing.

The three arms

The arms are a controlled experiment: same scenario, same model, different amounts of the Gettier machine.

• raw — the baseline. One direct model call, forced to emit a fix (propose_fix tool), with no declaration and no sensors. What the model does with no grounding at all.
• gated — the product. The in-process turn state machine with real sensors against the live fixture: declare → verify → gate → ground. If the gate holds, no fix is authored. The run's audit chain is persisted (and hash-verified) as evidence under runs/ledgers/.
• static — the control. The same measured sensor readings the gated arm would collect, handed to the model as framed context — but with no declaration, no gate, no verify loop. This isolates the verification loop from mere information: if the static arm still lucky-passes while the gated arm catches the fault, the gate — not the data — is the active ingredient.

Why the static arm matters

"You just gave the model more context" is the obvious objection to the whole thesis. The static arm is the answer to it: it gives the model exactly that context and nothing else.

Scoring: the four outcomes

Every run lands in one of four buckets (scenarios/SPEC.md), decided in deterministic core code from the seeded false lemma, the gate decision, and the acceptance result:

Verdict	Meaning
knowing_pass	Acceptance passed and the seeded false lemma's domain was declared and verified — right for the right reason.
catch	The gate stopped the false premise before a fix shipped (contradicted, or held on the lemma's domain). The win condition.
lucky_pass	Acceptance passed while the seeded false lemma went undeclared or unverified — right by luck. This is the failure the whole product exists to count.
miss	Acceptance failed, the model refused, or its fix could not be parsed. (Refusals are reported distinctly from parse-failure misses.)

A catch is decided before the harness looks at whether a fix was submitted — stopping a false premise is the win regardless of what came after. So a catch does not imply a grounded fix was applied, and the published table counts those runs separately rather than folding them in. The same serialization failure scores miss in the raw arm, where there was no gate to have caught anything; the arms are not symmetric on this point and the table says so.

Publishing: what refuses to emit a table

pnpm --filter @gettier/harness publish-benchmark writes runs/BENCHMARK.md, and it declines rather than qualifies. Four gates, each of which exists because its absence produced a table that had to be withdrawn:

1. Breadth and contamination — assessCoverage(): 12–16 scenarios, ≥20% holdout, and a MEASURED trap shape for every scenario (unclassified blocks; an unproven trap cannot back a published number).
2. The control arm — assessControlArm(): each scenario's raw arm must spring its own trap in at least 1/3 of runs. A gated catch fires before the acceptance test, so a scenario the model could not complete unaided would catch every gated run while proving nothing. GTR-008 caught 12/12 and read as the strongest scenario in the set while its control arm sprang 1 in 16 — its acceptance test asserted a response key the task never stated, so it was measuring vocabulary. Four scenarios have been repaired because of this floor. harness trap-check does not substitute for it: trap-check validates the one naive fix a fixture ships, not the space of fixes a model actually writes.
3. Scenario versions — runs carry detail.scenarioVersion, and runs produced by a superseded definition are EXCLUDED, not averaged. SPEC says a change to setup, prompt or test bumps the version; without the field, results from two definitions of the same scenario pooled silently.
4. Completeness — a run whose arm throws stores nothing, so the matrix quietly comes up short. The harness collects arm failures, groups them by cause, and exits non-zero. Both API accounts once ran out of credits mid-matrix: 42 of 312 runs never happened, three scenarios ended up with all their gated runs and none of their raw ones, and every scenario still reported success.

The table it emits reports the holdout and training strata separately, fix-delta separately from blind-window, and the per-scenario control-arm rates — see scenarios/SPEC.md for why each split must not be collapsed.

Multi-run variance

--runs N repeats each cell N times. With N > 1 the read models report per-cell variance — a cell (one scenario × model × arm) is unanimous when every replay agreed and split when they diverged — plus the seeded-declaration rate: the fraction of runs in which the model re-declared the same seeded false lemma. A single run (n=1) is an anecdote; a stable verdict across replays is evidence that the gate is deterministic, not lucky. The Scenarios drill-down in the web app surfaces this per cell.

Where the evidence lands

• A portable JSON report per run in runs/.
• For gated runs, the full per-run audit chain under runs/ledgers/ — the chain is the evidence, so it is persisted, not deleted.
• Every run is also written through the tenant resolver into the --data-dir ledger, which is exactly what the dashboard reads. Point the web app at the same GETTIER_DATA_DIR / GETTIER_TENANT to browse Scenarios, Replays, Lucky passes, and Model profiles.