I built semantic code tools for my AI agent. It ignored them.
A controlled experiment, a falsified hypothesis, and what it taught me about building tools for agents instead of people.
First, a disclosure that will be a fixture on this blog: I build software the way most working engineers do now — directing an AI coding agent, reviewing its work, making the calls it shouldn’t make alone. When I write “I” here, that’s shorthand for that partnership: my judgment and decades of .NET scar tissue, its speed. I’m not going to pretend otherwise, and I’m not going to write “my agent and I” four hundred times either. This is the new workflow. This blog is largely about it.
Which makes this particular story almost uncomfortably recursive: it’s about tools my agent and I built for agents — that agents then declined to use.
The itch
AI coding agents navigate C# codebases with text search. Grep finds strings; it doesn’t
find symbols. Search for references to a method named Save and you get every
Save in the codebase, plus comments, plus string literals. The compiler knows better —
Roslyn, the .NET compiler platform, can answer “who actually calls this Save”
precisely.
So I built RoslynMcp: an MCP server exposing semantic tools to Claude —
find_references, go_to_definition, find_implementations, symbol_search,
get_diagnostics (compile errors without a build), and a solution-wide
rename_symbol_preview. Nine tools, integration-tested against a real codebase with
known-true answers. The tests don’t check that the tools return plausible JSON; they
check that find_references names exactly the three classes I know consume a
particular component. Green means correct.
Then I wired it into my agent sessions and added guidance to the system prompt: prefer these over text search on C#; load the solution first; here’s how to chain them.
And then I did the thing that separates a weekend project from a product: I checked whether the agent actually uses it.
The experiment
Setup: Claude (Opus) in headless mode, on a real ~100-file C# codebase, with RoslynMcp as the only MCP server and the system-prompt guidance in place. The task was deliberately navigation-heavy and deliberately neutral — trace a feature’s lifecycle across the codebase and document it with verified line references. The task never mentions the tools. If the guidance works, it has to work on its own.
Run 1:
| Tool | Calls |
|---|---|
| Read | 10 |
| Bash (mostly grep) | 9 |
load_solution |
1 |
find_references, symbol_search, everything else semantic |
0 |
It loaded the solution — the cheap step the prompt recommended — and then did every piece of actual work with Read and grep.
The trust hypothesis
Before concluding “agents just won’t adopt tools,” I looked at what load_solution
had returned. Buried in its response was this:
"Failure: Msbuild failed when processing the file '...' with message:
Package 'SQLitePCLRaw.lib.e_sqlite3' 2.1.11 has a known high severity vulnerability"
A NuGet security advisory — one with no patched version upstream, surfacing through the design-time build as a raw “Failure” string. The solution had loaded fine. Every project was navigable. But the tool’s own response said failure, and the agent did something entirely rational: it stopped trusting the index and fell back to the tool it could verify.
That’s a hypothesis worth money: agents abandoned the tools because the tools
announced themselves as broken. So I fixed it, twice over. The advisory got a
targeted suppression at the source (it was noise — unfixable upstream, irrelevant to
navigation). And load_solution now states its status in plain language: “Solution
loaded successfully. Navigation tools are ready to use.”
Run 2 — same task, pristine load response:
| Tool | Calls |
|---|---|
| Read | 12 |
| Bash + Grep | 11 |
load_solution |
1 |
| Everything else semantic | 0 |
Identical behavior. Loaded the solution, never touched navigation, completed the task with Read and grep. Hypothesis falsified. The scary diagnostics were real polish worth fixing — but they were never the reason.
What I now believe
Advisory prompting produces token compliance, not adoption. The agent did the one
cheap thing the guidance recommended — load_solution — and then reverted to trained
habit for the real work. It followed the letter of the instruction and none of its
spirit, which is a very familiar failure mode to anyone who has ever managed humans.
Habits only lose when they fail. At ~100 files, reading everything still works. The agent’s grep-and-read instincts have been reinforced across effectively every codebase it has ever seen. A paragraph of prompt guidance was never going to outweigh that — not while the habit kept succeeding.
Trust is still a product surface. Run 1’s lesson stands even though it wasn’t the primary cause: one scary-sounding-but-benign message in a tool response can zero out an agent’s willingness to rely on it. If you build MCP tools, the emotional tone of your diagnostics is part of your API.
What I’d tell anyone building MCP tools
- Build capability-gap tools, not efficiency tools. Agents reach for a tool when the alternative is impossible, not when it’s merely slower. Grep cannot apply a solution-wide rename. Grep cannot compute type hierarchies. That’s where the adoption is. My v2 plans changed accordingly.
- Measure adoption; don’t assume it. Two transcript greps told me more than any amount of “this should help” reasoning. Total cost of the experiment: $3.62.
- Say plainly whether your tool worked. Structured status beats raw diagnostics. An agent parses “loaded successfully, navigation ready” very differently from a wall of “Failure:” strings — even when they describe the same state.
- Untested levers remain: directive prompting (“always use find_references for C# reference tracing” instead of “prefer”), workflow-level integration (hooks and skills rather than in-context choice), and genuinely large codebases where read-everything finally breaks down. If you’ve tested any of these, I’d genuinely like to hear what you found.
The tools remain installed. The tests stay green. And somewhere in my repo, nine semantic tools wait patiently for either a bigger codebase or a better argument — whichever arrives first.
Built and written in collaboration with Claude (Anthropic’s Fable 5) — the workflow this blog is about. The tool, its answer-key test suites, and the full experiment writeup are at github.com/sharpdaddy59/RoslynMcp; the .NET library for driving Claude Code sessions is at github.com/sharpdaddy59/ClaudeCode.NET.