We Ran a Three-Arm Benchmark. We Lost On Per-Query Tokens. Here's Why We Still Win.

We installed all three tools, indexed the same codebase, ran five identical queries, and measured everything. Here are the real numbers — including the parts where we lose.

The Setup

We installed all three tools — grep, Synapse MCP, and codebase-memory-mcp — on the same machine. Indexed the same codebase (our own — 220 Elixir source files, 64,184 lines, 5,532 indexed chunks). Ran five queries that represent what an AI coding agent actually does all day:

1. Find security-sensitive function calls (File.write, File.read, System.cmd)
2. "Who calls get_chunk?"
3. "Can I delete this function?"
4. "What does this codebase look like?"
5. "What breaks if I change store/ets.ex?"

Then we measured everything. No cherry-picking.

The Per-Query Numbers

Shell is fastest. CBM produces the most compact payloads. On per-query token count, we came second.

On per-query token count.

Why Per-Query Tokens Are the Wrong Metric

Here's the thing about compact payloads: they're compact because they contain less context.

When CBM returns 20 function names with zero callers in 105 tokens, that's efficient. But it's also incomplete. The agent now needs to follow up: "Is parse_sse_block an entry point or genuinely dead?" "Is findPidByPort called from a test or production?" "Is defaultXdgConfigDir public API?"

Each follow-up is another tool call, another round trip, another payload. The 105-token answer just cost you 5 follow-up queries at 500 tokens each. Total: 2,605 tokens to answer one question.

Synapse answered the same question in one call — 526 tokens — with safe_to_delete: false, confidence: low, production_callers: 0, and agent_instruction: "Do not delete source code from this result; resolve the exact symbol first." The agent has a verdict. It moves on. No follow-up queries needed.

The metric that matters isn't tokens per query. It's total tokens to reach a correct answer. Richer context per query means fewer queries. Fewer queries means fewer round trips, less model inference, and less total token spend — even if each individual payload is larger.

And Synapse gives the agent control over how much context it receives. Request format: "outline" to strip function bodies and get signature-only views. Set max_tokens to cap response size. Set compress_payload: false when you need full detail. The agent chooses how much to spend — it's not locked into one payload size.

What the Numbers Don't Show: Total Session Cost

The per-query table above is misleading because it measures isolated tool calls, not full agent sessions. Here's what a real session looks like — our existing security audit case study, same codebase, same model (gpt-5.5), full autonomous run:

177K tokens with shell vs ~71K with Synapse — across the full session. Half the tool calls. Twice as fast. Same vulnerabilities found (plus one dependency vuln that shell missed).

That's the number that matters. Not tokens per query — tokens to finish the job.

The Query That Breaks the Other Tools

"Who calls get_chunk?"

Shell returned 140 grep lines. Every mention of "get_chunk" — comments, test mocks, variable names, a docstring that says "gets the chunk from ETS." The agent gets 4,746 tokens of noise and has to read every line to figure out which are real calls.

Synapse returned 4 callers. Each with confidence: medium, edge_type: calls, file path, and line number. Zero false positives. 353 tokens. The agent knows exactly what calls what and acts immediately.

CBM returned 72 callers across multiple hops. More breadth, no confidence scores. 967 tokens.

Shell gives you the most data. Synapse gives you the most precise data. CBM gives you the broadest data. But 140 lines of grep noise is how production breaks at 2am — the agent thinks it found all the callers, but half of them were comments.

"Can I Delete This Function?"

This is the query that separates the tools entirely.

Shell listed 50 defp declarations. It can find private functions. It cannot tell you if any of them are dead. 1,688 tokens of function names, no analysis.

CBM returned 20 zero-caller function names via Cypher. 105 tokens. Compact. But it's a list — no safety assessment, no confidence, no distinction between "this is an entry point that's supposed to have zero internal callers" and "this is genuinely dead." The agent needs 5-10 follow-up queries to determine what's actually safe to delete.

Synapse returned a structured decision:

is_dead: false
safe_to_delete: false
confidence: low
deletion_risk: high
production_callers: 0
agent_instruction: "Do not delete source code
from this result; resolve the exact
symbol first."

526 tokens. One call. A complete answer — verdict, evidence, confidence, risk, and next step. No follow-up queries needed.

This is the difference between a tool that gives you data and a tool that gives you a decision. One saves tokens per query. The other saves you from a 2am incident.

"What Breaks If I Change This File?"

Shell: 313 lines mentioning "Store.ETS." 10,769 tokens. Every line that contains the string. No impact analysis. Just text.

CBM: 0 impacted items. Its change detection only works on git diffs — things that already changed. Ask it "what would break?" and it has nothing to say.

Synapse: 10 blast radius nodes, each with risk_level and risk_score. Coverage maps. Blind spots. 1,047 tokens. A structured impact assessment before you write a single line.

We're the only tool that tells you what will break before you break it.

The Real Cost: Total Session, Not Per Query

At frontier model pricing ($3–$15 per million tokens), over a 20-day work month — based on full-session token counts from our security audit benchmark:

Synapse Pro at $19/month pays for itself in the first week — not from smaller payloads, but from fewer queries, fewer round trips, and fewer tokens wasted on false positives and follow-up exploration.

What Only Synapse Has

• Safe writes — simulates edits in memory, verifies every caller, auto-rolls back if anything would break. Neither shell nor CBM can write files at all.
• Dead code decisions — structured safe_to_delete verdict with confidence and evidence. CBM gives you a list of names; Synapse gives you a decision.
• Change review — blast radius with risk scores before you push. CBM only works post-hoc on git diffs.
• Stack trace resolution — paste a crash trace, get the root cause mapped to your code. No other tool does this.
• Knowledge cache — learns from searches and persists across sessions. The tool gets smarter the more you use it.
• Worktree overlay — sub-50ms delta indexing for parallel agents in Git worktrees.

What CBM Does Better (Yes, Really)

• 158 languages vs our 50+. We add languages on request — if yours isn't supported, open an issue and we'll add it.
• Cypher queries — a full query language for ad-hoc graph pattern matching. We give you structured tools. Different philosophies.
• 3D visualisation — a built-in UI. We had it. We took it out. It was expensive and nobody used it more than once.
• 43 agent auto-config surfaces on install. We're catching up.
• CLI mode — every CBM tool works as a one-shot shell command. Ours are MCP-only.

CBM is a genuinely good tool. If you need breadth and query language power, it's excellent. But it's read-only. It can tell you who calls a function. It can't stop you from breaking those callers.

Why We Built This

Every tool we tried — CBM, Cursor's built-in indexing, Greptile, Augment — could tell us what was in the codebase. None of them could tell us whether it was safe to change it.

That's the gap. That's why Synapse exists.

We didn't build another code explorer. We built a tool that prevents the thing every developer has experienced: an AI agent that confidently refactors a function, updates 2 callers, declares success, and silently breaks 3 others in sibling packages that it never found because grep doesn't understand module aliases.

Safe writes. Dead code decisions. Blast radius before you push. Stack trace resolution. Persistent learning. These aren't features on a comparison matrix. They're the difference between shipping with confidence and shipping with fingers crossed.

Try It — Free, And You Get a Lot

The free tier isn't a trial. It's a full product.

You get AST indexing across 50+ languages. Three search modes — semantic, symbol, regex. File inspection with outline compression. Standard file writes. SmartCrusher payload compression. OS-level background file watching. Cross-repo edge resolution. Git worktree overlay with sub-50ms delta indexing. Unlimited repositories. No time limits. No credit card.

Pro adds the safety layer — safe writes with auto-rollback, crash resolution, change review with blast radius, test intelligence, dead code decisions, and persistent knowledge cache. $19/month. Pays for itself in the first week.

curl -fsSL https://downloads.synapse-mcp.dev/install.sh | sh

All benchmark scripts and raw outputs are published for reproducibility. Run it yourself. Prove us wrong.