Yesterday I found a bug that my test suite had been actively vouching for. Not missing — vouching for. The test asserted the exact behavior, the behavior was wrong, and the test passed anyway. It’s the most instructive bug I’ve hit in years, and it’s a class of failure worth naming.

The bug

I have an analyzer that reads legacy .NET solutions and computes a migration order from the project reference graph — leaves first, so shared libraries migrate before the things that depend on them.

Old-style csproj files store references as Windows-style paths:

<ProjectReference Include="..\InventoryCore\InventoryCore.csproj">

My code extracted the project name with Path.GetFileNameWithoutExtension. On Windows that returns InventoryCore. On Linux — where this analyzer runs — backslash isn’t a path separator, so it returns ..\InventoryCore\InventoryCore. Which matches no project name. Which means every reference silently failed to resolve. Which means every project looked like a leaf. Which means my “topological sort” was, in fact, an alphabetical one.

The lie

Here’s the part that stings. I had a test for exactly this:

[Fact]
public void MigrationOrder_IsLeavesFirst()
{
    Assessment.SuggestedMigrationOrder.First().Should().Be("InventoryCore",
        "both other projects reference it");
}

InventoryCore is the correct first answer — it’s the leaf. InventoryCore is also the alphabetically first project in the solution. The broken sort and the correct sort produce the same output on this input. The test stayed green through every commit that followed, not because the code was right, but because my test fixture couldn’t tell right from broken.

The bug surfaced the first time I pointed the analyzer at a codebase I didn’t author — a 17-project Codeplex-era app — and saw Terminals.Common scheduled for migration after the application that depends on it. A human glance caught in two seconds what the green suite had been happily certifying.

The class of failure

Call it right by luck: the test’s expected answer coincides with the degenerate answer. Every degenerate fallback has a signature — alphabetical order, empty collections, zeros, the input echoed back — and if your fixture’s correct answer matches the signature, your test is a rubber stamp.

The fix is adversarial fixture design: choose test data where the correct answer and the lazy answer visibly differ. My regression test now uses a project graph where Beta depends on Zed — alphabetical says Beta first; correctness says Zed. The degenerate path can never pass it again.

Three habits I’m keeping from this:

  1. When writing a fixture, ask: what would a broken implementation output here? If it’s the same as the right answer, change the fixture until it isn’t.
  2. Feed your tools at least one input you didn’t author. Synthetic fixtures catch regressions cheaply; wild specimens catch the failures you didn’t know to plant. You need both — my synthetic suite still runs on every commit, but it no longer testifies alone.
  3. Treat “it passed” with the same suspicion as “it works on my machine.” A test only means what its fixture can distinguish.

Nothing about this is new wisdom — mutation-testing people have preached it for decades. But there’s a modern reason to re-learn it: I build with AI agents, and agents generate plausible tests fast. Plausible is exactly the danger. A test suite that grows quickly is a test suite whose fixtures nobody stared at long enough to ask what a broken implementation would say. The staring is still our job.

Built and written in collaboration with Claude (Anthropic’s Fable 5) — the workflow this blog is about. The semantic code tools from post one: github.com/sharpdaddy59/RoslynMcp.