(As always: “I” means the partnership — my direction and review, the agents’ execution.)

Last time, I found that explicit acceptance criteria have a side effect I named the checklist effect: enumerate the edge cases and the strong models write the listed tests, satisfy them, and stop being curious. Sonnet’s test count fell from 146 to 96 when I handed it the answer key. I ended that post promising an escape hatch — a clause telling the model the list is a floor, not a boundary — and admitted I didn’t know whether it would work.

So: the same 8-feature calculator plan, the same explicit landmine-naming acceptance criteria, the same fixed reviewer (Claude Fable 5), plus one appended sentence per feature: “the enumerated cases are a floor, not a boundary — test beyond this list.” Registered predictions, filed before the run: Sonnet’s tests recover to 130+. Opus recovers partially, to 430+. Haiku barely moves (~115) because the mandate can’t buy capability. Rejections stay about where they were: 6 / 2 / 0.

The runs fired the night the weekly quota reset and finished in two hours. Here’s the complete grid, all three spec styles:

Tests written / rejections

  Vague Explicit Explicit + explore
Haiku 4.5 111 / 4 114 / 6 177† / 7 — DNF at 5/8
Sonnet 5 146 / 2 96 / 2 171 / 1
Opus 5 498 / 0 387 / 0 613 / 0

†Haiku’s 177 tests cover only the five features it completed before the pipeline’s stall detector gave up on it. DNF — did not finish — the first in nine runs.

Finding 1: The escape hatch works — the checklist effect was real, and it’s reversible

Sonnet: 96 → 171, past even its vague-spec 146, with rejections down from 2 to 1. Opus: 387 → 613, blowing through my “partial recovery” prediction and past its vague-spec 498. One sentence bought back all the curiosity the checklist had suppressed, and then some — because now the exploration starts from the floor of the enumerated cases instead of from scratch.

That’s the spec recipe finalized, as far as I’m concerned: name every landmine you know (the floor), then mandate exploration beyond the list (the ceiling). Explicit floors prevent the defects you can foresee; the exploration clause hunts the ones you can’t. For capable models it’s the best cell in the grid on every axis.

Finding 2: My first DNF — and the prediction I’m glad I registered

For Haiku I registered not just a prediction but its interpretation, in advance: if Haiku’s test count jumps, then the checklist effect was instruction-following rather than native thoroughness. Its test count jumped — 177 tests, more than Sonnet, on five-eighths of a project.

Haiku obeys. That’s the finding. Told to explore beyond the list, it diligently wrote half again as many tests — and the reviewer rejected its features seven times anyway, because the tests weren’t the ones that mattered. The StackOverflow-on-deep-nesting trap it has now failed in three consecutive benchmarks. Mandatory CLI tests that two rework cycles never produced. And then the failure mode that actually killed the run: a fix session that lasted seven seconds — long enough to look at plausible code and green tests and conclude “already done,” despite the rejection reason sitting right there in its context. Two consecutive no-progress sessions and the pipeline’s stall detector called it.

Obedience is not capability. The exploration mandate is an amplifier, same as the explicit spec was: it hands capable models a license and weak models a rope.

I don’t publish an n=1 DNF, so before writing this up I reran the Haiku arm from scratch. Run B: 5 of 8, 7 rejections, stalled at session 6 — the identical endpoint, down to the same three features left unfinished (evaluator, REPL, CLI polish). The specific rejection reasons were different this time — a tokenizer happily accepting Unicode digits, wrong-arity errors that don’t name the function — and it wrote even more tests (256). Different potholes, same ditch. The DNF isn’t an unlucky roll; it’s the reproducible place where this model meets this spec.

Finding 3: The money now tells you where the effort went

Cost per approved feature Vague Explicit Explore
Haiku $1.44 $1.71 $2.74 (5 features)
Sonnet $2.11 $2.43 $2.26
Opus $2.63 $2.45 $3.37

Haiku has lost the cheapest-per-feature crown that justified its existence in this series — and the detail underneath is worse: the reviewer spent $9.42 refuting Haiku’s work while Haiku spent $4.30 producing it. When your review bill is 2.2× your implementation bill, the model tier is below the task; I’m adopting that ratio as an early-warning gauge.

Sonnet’s exploration run cost less than its checklist run ($18.11 vs $19.42 all-in) while writing 78% more tests. Free quality. Opus’s $26.94 is the priciest run of all nine — 613 tests aren’t free — but every marginal dollar went into implementation depth rather than review friction.

The series verdict in one line: spec style doesn’t change what a model can do — it changes where the money goes. Vague specs spend it on reviewer detective work. Explicit specs spend it on reviewer enforcement. Exploration mandates spend it on implementer depth — which is only a good trade if your implementer can convert depth into approved features.

The usual honesty

Still n=1 per cell — except the DNF, which earned itself an n of 2 and reproduced exactly. Still one domain, one reviewer as the measuring stick. Scorecard on the registered predictions: Sonnet confirmed, Opus exceeded in the direction I hedged against, Haiku wrong in precisely the way I pre-committed to interpreting, and the rejection line missed in both directions at once. Two and a half out of four. The half is doing a lot of work.

Previous entries: the tier benchmark, the $8 code reviewer, and I Was Wrong About Specs.