(As always: “I” means the partnership — my direction and review, the agents’ execution. — and this time the models were also the test subjects)

Last year I built SharpCoder, a tool of mine that drives autonomous Claude Code sessions to build applications feature-by-feature. You give it a feature plan; it implements each feature, runs the build, runs the tests, and then — this is the important part — a separate Claude session with fresh context adversarially reviews the work before anything counts as done. The reviewer’s instructions boil down to: assume it is wrong, try to break it, reject with a specific reason if you succeed.

This weekend I added run instrumentation to SharpCoder: a sessions table, durable rejection counts, test totals captured from the gates, honest cost accounting from the Claude Code transcripts. And the obvious first experiment for a freshly instrumented pipeline was one I’d wanted to run properly for months:

Same plan. Three model tiers. Fixed judge. What do you actually get for your money?

The setup

  • One fixed plan: an 8-feature command-line expression calculator (tokenizer, parser, evaluator, variables, built-in functions, REPL, CLI). Classic kata territory: objectively testable, and absolutely riddled with edge cases.
  • Three runs overnight, identical except for the implementation model: Claude Haiku 4.5, Claude Sonnet 5, and Claude Opus 5, each building the whole project from scratch in its own directory. (I’ll call them Haiku, Sonnet, and Opus below.)
  • The same reviewer for all three: Claude Fable 5 (Anthropic’s top-tier model) adversarially reviewed every feature in every run. Fixed judge, fixed bar.
  • Same prompts, same gates, same everything else.

One design decision mattered more than any other — and it came from a different project entirely. A day earlier, I’d had SharpCoder build an RFC 5545 recurrence-rule engine from a plan whose acceptance criteria named the landmines explicitly (“months lacking day 31 are SKIPPED — no Feb 28 substitution”). The result was zero rejections across 17 features: the model wrote tests for exactly the traps the spec enumerated and never stepped on one. Great for production. Useless for a benchmark — enumerate the edge cases and you’re measuring your spec, not the model.

So this plan states outcomes the way a normal user would: “standard mathematical precedence and associativity,” “invalid input produces a clear error, never a crash.” It never mentions that ^ is right-associative, never asks what -2^2 should mean, never dictates division-by-zero semantics. The model’s native thoroughness is the variable under test — and the reviewer’s rejection count is the measuring stick.

The results

  Haiku Sonnet Opus
Review rejections 4 2 0
Implementation sessions 5 2 1
Active time 29 min 32 min 45 min
Tests written 111 146 498
Complexity (avg / max) 1.81 / 14 2.13 / 20 2.85 / 14
Cost: implementation $3.70 $9.68 $16.62
Cost: review $7.82 $7.16 $4.45
Cost per approved feature $1.44 $2.11 $2.63

(Costs are API-list-price equivalents computed from the session transcripts. I run all of this on a Claude Max subscription, so my actual marginal cost was $0 — but the equivalents are what make tiers comparable.)

All three runs finished with 8/8 features approved. That sentence hides everything interesting.

The rejection gradient: 4 → 2 → 0

The gradient itself was predictable. What I didn’t predict was how cleanly the character of the failures would sort by tier.

Haiku’s four rejections were shallow gaps — the “you stopped too early” category:

“Parser.Parse() lets a raw System.OverflowException escape, violating ‘never an unhandled exception’.”

“The Infinity/NaN check was added to unary and binary results only; function-call results are still unchecked.”

That second one is the signature Haiku failure: it understood the requirement, implemented it, and applied it to two of the three places it mattered. The third rejection was simply missing tests the acceptance criteria required.

Sonnet’s two rejections were genuinely deep — bugs plenty of human reviewers would miss:

“Deeply nested input crashes the whole process with an uncatchable StackOverflowException and a raw stack trace.”

“An overflowing number literal evaluates to raw Infinity instead of an error: qcalc \"1e999\" prints ‘Infinity’.”

A recursive-descent parser that dies on ((((((...)))))) is a real production bug class. Sonnet cleared the bar Haiku tripped on and hit the next one up.

Opus was rejected zero times. And the table shows how it bought that: 498 tests, 4.5× what Haiku wrote. Opus spent its extra fifteen minutes making itself confront its own edge cases before the reviewer could. By the time Fable got there, the traps had already been sprung — by Opus, on Opus.

The money table is sneakier than it looks

Haiku won cost-per-approved-feature — $1.44, despite four rework cycles. Five-times-cheaper tokens absorb a lot of rework. If your definition of done is “the reviewer approved it,” the budget tier plus an adversarial gate is a legitimately good deal.

But two things complicate the victory lap.

First: 68% of Haiku’s bill was the reviewer. And look at the review row — it falls as the implementer improves: $7.82 → $7.16 → $4.45. Bad code is expensive to review. More findings to verify, more re-reviews after fixes. The reviewer is cheapest exactly when it’s needed least.

Second: the runs didn’t ship the same product. All three calculators are “approved,” but one comes with 111 tests and another with 498. Ask me which codebase I’d rather extend next year.

The verdict matrix

My takeaway, compressed:

  • Vague spec + unfamiliar domain → the tier gap is real. Pay for the top tier or budget for rework cycles.
  • Tight spec with the edge cases named → cheaper tiers plus an adversarial review gate punch far above their weight. (That’s the recurrence-engine result from earlier: 17-for-17, zero rejections — but arguably the spec earned that score as much as the model did.)
  • The adversarial reviewer is worth it at every tier. It’s the fixed instrument that makes any of this measurable — and it costs single-digit dollars per project. That economics story is its own post: The $8 Code Reviewer.

Caveats, because this is a case study and not science

One run per tier. One domain. The reviewer’s approval is a floor, not proof of correctness — bugs below Fable’s detection threshold are invisible to all of these numbers. Wall-clock comparisons are polluted by rate limits (Sonnet’s run paused for an hour mid-flight; the table shows active time). Different plans would sort differently.

But n=1 with instrumentation beats n=0 with opinions, and every number in this post came off a dashboard instead of out of my memory. SharpCoder is a private project for now — but the experiment design (fixed plan, fixed adversarial reviewer, vary the implementer) is reproducible in any agent harness with a few gates.

Built by one retired guy and a lot of autonomous Claude sessions. The models built the calculators; a model reviewed the models; I supervised from the couch with a beer.