Results¶
384 graded runs. 16 scenarios. 4 frontier models. Two arms each, three repetitions.
Lucky passes fall from 96% of runs to 0% — 184 to zero — and the gate contradicts the seeded false premise in 100% of gated runs against 0% unaided.
Every model, same trap¶
Each model ran every scenario twice: once on its own, once with Gettier in front of it. Same model, same task, same containers.
| Model | Shipped a fix on a false premise | Failed outright | Contradicted before the fix was written | |
|---|---|---|---|---|
gpt-5-mini |
on its own | 48 of 48 — 100% | 0 | — |
| with Gettier | 0 of 48 — 0% | 0 | 48 | |
gpt-5 |
on its own | 47 of 48 — 98% | 1 | — |
| with Gettier | 0 of 48 — 0% | 0 | 48 | |
claude-sonnet-5 |
on its own | 47 of 48 — 98% | 1 | — |
| with Gettier | 0 of 48 — 0% | 0 | 48 | |
claude-haiku-4-5 |
on its own | 42 of 48 — 88% | 6 | — |
| with Gettier | 0 of 48 — 0% | 0 | 48 | |
| All four models | on their own | 184 of 192 — 96% | 8 | — |
| with Gettier | 0 of 192 — 0% | 0 | 192 |
"Shipped a fix on a false premise" is a run that passed its acceptance test while the load-bearing false premise went undeclared or unverified. Green tests, working code, rotten reason. In the scoring taxonomy it is a lucky pass — the failure no outcome benchmark can see, because an outcome benchmark only asks whether the task passed.
"Contradicted before the fix was written" is a catch: a sensor measured the real system, the measurement disproved the premise, and the gate stopped the change while it was still a plan.
The lowest number in that column is not the safest model
claude-haiku-4-5 lucky-passes least often — 88% — because it fails outright more
often: 6 of its 48 unaided runs missed, against 1 for the two larger models. It is not
catching the trap. It is falling over before it gets there, which is the honest failure
mode and the one you would rather have.
Read the column as how often a model sails past a false premise it never noticed, not as a ranking of quality.
This is not a weak-model problem¶
The spread between the strongest and weakest model here is twelve points, and the floor is 88%. Nothing in this table suggests the next model generation solves it: a better model reasons better from the premises it has, and the premise these scenarios falsify is one the model cannot see from the code — the reverse proxy's cap, the database it is actually connected to, the runtime production runs.
That is the argument for measurement rather than capability. The gated column is not a smarter model; it is the same model with its assumptions checked against the running system.
What the scenarios are¶
Sixteen containerised environments in which the plausible fix is wrong — and wrong in the way that matters, because it works on the test payload. The seeded fault is masked by something outside the diff under review.
6 of the 16 (38%) are holdout: declared in the specification before they existed, so no judgment rule was written against their answers. They are the check on whether the gate generalises or has merely been fitted to what it was built on.
| Stratum | Scenarios | Runs | Unaided lucky | Gated lucky |
|---|---|---|---|---|
| training | 10 | 120 | 95% | 0% |
| holdout | 6 | 72 | 97% | 0% |
Contamination would show up as a holdout rate worse than training. There is none here. That is evidence, not proof — six scenarios is a small stratum, and the honest reading is that this corpus no longer contains a masking type the gate misses, not that none exists.
What the table does not claim¶
- Not a general model benchmark. These scenarios are built to be traps. A 98% lucky-pass rate is a statement about deliberately masked premises, not about how often a model is wrong on ordinary work.
- A catch is not a shipped fix. 14 gated catches recorded the contradiction — the premise
never shipped, which is what
catchscores — but the model then failed to produce a well-formed fix. Counted separately rather than folded into the totals. knowing_passis unreachable on this corpus, so SPEC'slucky / (lucky + knowing)ratio is degenerate and is not the headline. Lucky passes per run is reported instead, with the raw counts above so you can compute either.
What it took to be allowed to publish this¶
Four earlier versions of this table were withdrawn. Each withdrawal became a gate that now blocks publication — breadth and contamination, a control arm that must spring its own trap unaided, scenario versioning, and completeness. The harness declines to emit a table rather than qualifying one.