(As always: “I” means the partnership — my direction and review, the agents’ execution.)

Two days ago I benchmarked Claude’s model tiers against a deliberately vague spec and got a tidy rejection gradient: Haiku 4, Sonnet 2, Opus 0. The obvious follow-up question — the one with actual money riding on it — was whether a better spec closes that gap. My pipeline had already produced a hint: an earlier project whose acceptance criteria named every edge-case landmine explicitly sailed through review with zero rejections. “Better plan = better code,” I figured. Write the edge cases into the spec, and maybe the budget model plus a good adversarial reviewer equals the premium model at a fifth the price.

So I ran the missing arm: the same 8-feature calculator plan, same three implementation tiers, same fixed reviewer (Claude Fable 5) — but with the acceptance criteria rewritten to name every trap. Right-associative ^. -2^2 = -(2^2). Division by zero must error, never print Infinity. Deep nesting must not stack-overflow. I even promoted the previous run’s actual rejection reasons into the spec: “the same checks apply to unary, binary, AND function calls — checking two of three is the classic bug” is literally Haiku’s own bench1 rejection, handed back to it as instructions.

I registered predictions beforehand, so I don’t get to quietly revise them: Haiku would drop from 4 rejections to 0–2. Sonnet from 2 to 0. Opus stays 0.

The grid

  Haiku 4.5 Sonnet 5 Opus 5
Rejections — vague → explicit 4 → 6 2 → 2 0 → 0
Tests written — vague → explicit 111 → 114 146 → 96 498 → 387
Cost per approved feature — vague → explicit $1.44 → $1.71 $2.11 → $2.43 $2.63 → $2.45

Haiku’s rejections went up. With the answer key in hand. Two of my four registered predictions died on contact with the data, and the two survivors were the boring ones.

Finding 1: An explicit spec doesn’t rescue the implementer — it arms the reviewer

Here’s the mechanism I missed. Every landmine you name in an acceptance criterion becomes a binary, checkable claim. Under the vague spec, my reviewer had to judge Haiku’s work against a reasonable interpretation — and reasonable interpretations leave room for mercy. Under the explicit spec, there’s a list, and the reviewer checks the list. Haiku’s rejections read like a failed exam against a posted rubric: token positions wrong whenever input contains whitespace; two rounds of unary-minus grammar defects; a division-by-zero check implemented as a hand-rolled |r| < 1e-14 epsilon that nobody asked for.

The strong model uses an explicit spec to pre-empt rejections — it writes the named tests first and walks through clean. The weak model can’t fully execute the same spec even when it’s handed the answers, and now every shortfall is objectively provable. Same document, opposite effect. Spec quality amplifies whatever capability is present — in both directions.

But before “better plan = better code” goes in the bin: look at what shipped. The explicit-spec calculator from Haiku, after its six rework cycles, has every named trap verified — it’s a genuinely better artifact than its vague-spec cousin, which merely satisfied a lenient reading. The refined verdict: the spec sets the bar; the model tier determines how many attempts it takes to clear it. You always get the bar you wrote. You pay for the attempts.

Finding 2: The checklist effect — explicit specs made the strong models less curious

The finding I didn’t see coming at all: Sonnet’s test count fell from 146 to 96. Opus’s fell from 498 to 387. Enumerated acceptance criteria appear to work like a checklist — write the listed tests, satisfy them, stop. The vague spec, by giving the stronger models nothing to check off, forced them to ask “what could go wrong here?” — and Opus answered that question 498 times.

So explicit specs raise the floor and quietly lower the ceiling. They guarantee the cases you thought of and mildly discourage hunting for the ones you didn’t. If you want both — and for anything that matters, you do — the spec probably needs its own escape hatch: an explicit “and test beyond this list; the enumerated cases are a floor, not a boundary.” Whether that clause actually restores the curiosity is an empirically checkable question, which is exactly what my next run is for.

Finding 3: With a good spec, the premium model costs the same as the mid-tier

The economics table quietly rewrote my production recipe. Under the explicit spec, Sonnet cost $2.43 per approved feature and Opus cost $2.45 — two cents apart. The explicit spec made Opus cheaper (fewer speculative excursions, zero rework) and made Haiku pricier (rework cycles, and every rejection means the reviewer bills again for the re-review). The tiers’ costs converge — and at converged prices, Opus delivered zero rejections to Sonnet’s two and four times the tests.

My earlier post concluded that the budget tier plus an adversarial gate was a legitimately good deal. That conclusion survives only for vague specs. Once you’ve invested in a real spec: explicit spec + top-tier implementer + top-tier reviewer = zero rework at mid-tier prices. The spec work you were going to do anyway pays for the model upgrade.

The usual honesty

Still n=1 per cell. Still one domain. Still one reviewer as the fixed measuring stick, whose approvals are a floor and not a proof. And two of my four registered predictions were wrong, which I’d rather publish than polish away — a benchmark that only ever confirms your expectations isn’t measuring anything.

The pipeline behind all this is a private project of mine; the experiment design — fixed plan, fixed adversarial reviewer, vary one thing at a time — ports to any agent harness with gates. Previous entries: the tier benchmark and the $8 code reviewer.