(As always: “I” means the partnership — my direction and review, the agents’ execution.)

I maintain RoslynMcp, an MCP server that hands an AI agent the C# compiler platform: find-references, go-to-definition, call trees, and — the part I was proudest of — refactorings with a compile-before-write guarantee. Rename a symbol across a solution, change a method signature at every call site, move a type to another namespace. Every edit is compiled in memory first; if it would introduce a new error, nothing touches disk and you get the exact diagnostic back.

Version 2.2 just shipped with 27 tools and a Claude Code plugin. So I did the obvious thing: wired it into SharpCoder, my autonomous coding pipeline, and measured whether agents actually use it.

They mostly didn’t. And after three experiments designed to make them need it, I think they were right.

The setup

SharpCoder builds and changes .NET code through autonomous Claude Code sessions, one feature at a time. After every session it runs the build and the test suite itself (a broken build reverts whatever the session claimed), and a separate, fresh-context reviewer session adversarially checks each feature before it counts as done (that gate has its own post).

For these experiments I added a --roslyn on|off switch to the headless runner, a per-session tally of every tool call, and an archive of each run’s transcripts. Then I ran A/B pairs: identical frozen copies of a codebase, identical plan, identical models — the only difference being whether the Roslyn server existed.

Experiment 1: the reviewer that didn’t need help

First, the cheapest test: replay a review on frozen code with known bugs. In July a Haiku implementer built a 10-feature reporting CLI that compiled, passed all 93 tests, and did nothing — Program.cs checked its arguments and returned 0 without ever calling the services. I froze the repo at that commit and ran one Opus review session per arm.

The Roslyn arm made zero Roslyn calls. The whole project was about 20 source files; the reviewer read them all in two cat commands, then ran the real binary against real databases. Both arms caught every known defect — plus one July’s reviewer had approved (the HTML report was never written either). Each review took about three minutes and under two dollars.

Fair enough: on 20 files, reading everything is the efficient strategy. Semantic tools need a codebase too big to read.

Experiment 2: five refactors on a real codebase

So I turned the pipeline on itself: a frozen copy of SharpCoder — about 126 code files, 16,000 lines — and a plan of five pure refactors, each picked to match a Roslyn write tool:

Refactor The Roslyn tool built for it
Rename AgentOrchestrator → AgentSessionRunner (class, interface, files) apply_rename
Move the build/test output parsers to a new namespace move_type_to_namespace
Extract an interface and switch consumers to it extract_interface
Thread a CancellationToken through a method and its callers change_signature, find_callers
Remove dead code from the core library find_unused_members

Sonnet implemented, Opus reviewed. Results:

  Roslyn off Roslyn on
Features completed 5/5 5/5
Build-gate reverts 0 0
Review rejections 0 1
Roslyn calls — 6, all analysis
Cost / wall time $3.50 / 12 min $4.84 / 9 min

Refactors 1–4 came out identical in both arms, and correct. The implementers did every rename and move with git mv and sed, then let the build tell them what they missed. Not one write-tool call.

The fifth refactor is where the arms split — and where Roslyn finally mattered, as an answer rather than an edit. The Roslyn-arm reviewer ran find_unused_members, found three unused constructors the implementer had kept, and rejected:

“…the dead-code sweep is incomplete: the semantic scan (roslyn find_unused_members on SharpCoder.Core) plus git grep shows at least one member that is unambiguously unreferenced…”

That rejection cost an extra fix-and-re-review round — the whole cost difference. Meanwhile the no-Roslyn reviewer answered the same question with homemade grep loops: three of them ran about 100 seconds each, 5.5 minutes of shell time in a 7.5-minute session. The Roslyn call took about ten seconds. (The no-Roslyn implementer, amusingly, wrote its own little Roslyn scanner from scratch — the compiler platform was already a dependency of the codebase.)

Experiment 3: refactors designed to break sed

Unique names are easy mode. So I designed four renames where blanket text replacement is wrong:

  1. Rename Stats on one of three view models that all have a Stats property — with XAML binding to all three by string.
  2. Rename SendMessageAsync on one chat service, while a second service and both view models’ command methods share the name.
  3. Rename one overload of an overloaded method, in the interface, the implementation and two test doubles.
  4. Rename FeatureStats.Total to FeatureCount — while Total also lives in a test-summary tuple, in log message templates, in anonymous types, and in a record that serializes the JSON an MCP tool returns to the agent.

That last one is the nasty one. Rename the record’s Total and the JSON contract silently changes — the build passes, and the only test round-trips the JSON through the same renamed record, so every test passes too. Only judgment catches it.

I ran it with Sonnet, then again with Haiku, Roslyn off and on. Four runs:

Implementer Roslyn Correct Rejections Roslyn calls Cost
Sonnet off 4/4 0 — $2.24
Sonnet on 4/4 0 0 $2.45
Haiku off 4/4 0 — $1.60
Haiku on 4/4 0 3 $1.66

All four runs produced byte-identical diffs. Every trap was dodged in every run, including the silent JSON one.

Here’s how Sonnet did it, in its own words from the lessons-learned file it keeps:

“Line-targeted renames (not a blanket sed): Total is shared with…”

It grepped for every occurrence, read each one in context, and ran sed against specific lines of specific files. It was doing a semantic rename by hand. When one of its regexes missed an overload call site (globalSettings with a capital S), it spotted the gap reviewing its own diff — and had it not, the build would have refused the missing method name anyway.

And the twist: the only agent in the whole study to pick up a Roslyn write tool was the weakest model. Haiku, on the Total rename:

“This one needs care: Total is also a tuple element, an anonymous/MCP JSON field, and log placeholders. A semantic rename is safer than text replacement, so I’ll load the Roslyn tools.”

It ran apply_rename as a dry run, applied it (seven files, that symbol only), and then — correctly noting that Roslyn doesn’t rewrite XAML — fixed the one binding by hand. Textbook. And the Haiku with no Roslyn produced the exact same diff with sed.

So what are refactoring tools for?

A refactoring tool does two jobs. It does the mechanics — finds every site, makes every edit. And it gives a guarantee — every reference updated, nothing else touched.

For a human in an IDE, the mechanics are most of the value. Editing 40 call sites by hand is slow and error-prone, and that’s exactly the work rename-refactoring deletes.

For an agent, the mechanics are nearly free. It doesn’t get bored or sloppy at call site 37; grep-read-edit across 40 files takes seconds. What it needs is the guarantee — and in these runs it got that from two places it already had. The compiler guards everything that would break compilation: missed references, a slipped overload call, XAML bindings (compiled bindings make those build errors too). The agent’s own reading covered what the compiler can’t see — the tuple names, the log templates, the JSON contract.

What an agent can’t cheaply produce for itself is knowledge: what’s unused, who calls this, what a change would touch and which tests reach it. Answering “is this used?” with grep means a loop over every identifier in the codebase. Answering it with Roslyn is one call. That’s the tool that got used, and the only one that changed an outcome.

So I trimmed SharpCoder’s prompts to match: analysis first, plus one line keeping apply_rename for names other symbols share — the single case where an agent actually reached for it.

Caveats, because this is still a case study

One or two runs per condition. A 16,000-line codebase — at 500 files per rename, or a signature change across a sprawling solution, grep-read-sed gets expensive and the balance could flip. These agents ran behind a build gate and a test gate; an agent without a compiler in the loop is in a different world. And this is C#, where the compiler is strict and fast; dynamically typed languages give text-editing agents far less of a safety net.

There’s also a deeper point I’m wary of: the agents had to choose to use the tools. Claude Code loads MCP tools lazily, so a tool the model doesn’t think to ask for might as well not exist. Some of “they didn’t need it” may be “they didn’t think of it.” But when Haiku did think of it, the outcome was identical — so I don’t believe I’m missing a hidden win.

Things I tripped over along the way

  • Claude Code deletes transcripts after 30 days by default. My July benchmark receipts were gone when I went looking. If you run experiments, raise cleanupPeriodDays or archive transcripts yourself — SharpCoder’s runner now does it automatically.
  • Plugins load in agent sessions too. With the RoslynMcp plugin installed, every autonomous session would have run two Roslyn servers — the plugin’s and the project’s. SharpCoder now disables the plugin per project.
  • The reviewer keeps getting better and cheaper. Today’s Opus caught a defect July’s reviewer approved, in about three minutes and under two dollars.

The takeaway

I set out to prove agents need a refactoring engine, and built traps to make sure. They walked around every one with grep, sed, a compiler, and careful reading — the same way they’d work in a language with no Roslyn at all. What they wanted from the compiler platform wasn’t hands. It was answers.

Built by one retired guy and a lot of autonomous Claude sessions. I built tools for the agents; the agents showed me which ones they wanted.