We Ran a Three-Arm Benchmark. We Lost On Per-Query Tokens. Here's Why We Still Win.

I benchmarked our own tool against the two alternatives a developer actually considers: plain shell tools (grep, find, cat) and codebase-memory-mcp (CBM). Same codebase, same queries, same machine. I'm going to show you the raw numbers — and I'm going to start with the one where we lost.

The setup

Three arms, all running on the same box:

• Shell tools — grep, find, cat. What every developer already has.
• Synapse MCP — our AST-based code knowledge graph server.
• codebase-memory-mcp (CBM) — a solid graph-based codebase indexer with Cypher query support.

The codebase was our own: 220 Elixir source files, 64,184 lines, 5,532 indexed chunks. I ran five queries that represent what an AI coding agent actually does all day:

• Find security-sensitive function calls (File.write, File.read, System.cmd)
• "Who calls get_chunk?"
• "Can I delete this function?"
• "What does this codebase look like?"
• "What breaks if I change store/ets.ex?"

I timed every call and counted every token. No cherry-picking, no reruns until it looked good.

The per-query numbers (where we lost)

Here's the table. Read it carefully — especially the token columns.

Let's be honest about what this shows. Shell is the fastest by a mile — 450ms total vs our 14.6 seconds. On raw token count, CBM wins with 2,591 tokens across all five queries. We came in second at 5,268. That's more than double what CBM returned.

We lost on per-query tokens. No spinning that.

But here's why that's not the whole story.

Why per-query tokens are the wrong number to optimize

CBM's payloads are compact. That's a real strength. But they're compact because they contain less context — and the agent has to go back and ask for the missing pieces.

When CBM returns 20 function names with zero callers in 105 tokens for the dead-code query, that's genuinely efficient as a single response. It's also incomplete. The agent now has to follow up: "Is parse_sse_block an entry point or genuinely dead?" "Is findPidByPort called from a test or production?" "Is defaultXdgConfigDir public API?"

Each follow-up is another tool call, another round trip, another payload. That 105-token answer just cost you 5 follow-up queries at 500 tokens each. Now you're at 2,605 tokens to answer one question.

Synapse answered the same question in a single call — 526 tokens — with safe_to_delete: false, confidence: low, production_callers: 0, and agent_instruction: "Do not delete source code from this result; resolve the exact symbol first." The agent gets a verdict. It moves on. No follow-ups.

The metric that matters isn't tokens per query. It's total tokens to reach a correct answer. Richer context per query means fewer queries. Fewer queries means fewer round trips, less model inference, and less total token spend — even if each individual payload is larger.

And we give the agent control over how much context it receives. Request format: "outline" to strip function bodies and get signature-only views. Set max_tokens to cap response size. Set compress_payload: false when you need full detail. The agent chooses how much to spend — it's not locked into one payload size.

What the per-query table hides: total session cost

The table above measures isolated tool calls. That's not how agents work. An agent runs a full session — dozens of queries, follow-ups, exploratory searches, and corrections. So I ran the real test: our existing security audit case study, same codebase, same model (gpt-5.5), full autonomous run with no human intervention.

177K tokens with shell vs ~71K with Synapse — across the full session. Half the tool calls. Twice as fast. Same vulnerabilities found, plus one dependency vuln that shell missed entirely.

This is where the per-query token argument flips. CBM won the per-query column. We won the session. And sessions are what you actually pay for.

The query that breaks shell tools: "Who calls get_chunk?"

This one is illustrative because it's the most common question an agent asks, and the three tools handle it completely differently.

Shell returned 140 grep lines. Every mention of "get_chunk" — comments, test mocks, variable names, a docstring that says "gets the chunk from ETS." The agent gets 4,746 tokens of noise and has to read every line to figure out which are real calls.

Synapse returned 4 callers. Each with confidence: medium, edge_type: calls, file path, and line number. Zero false positives. 353 tokens. The agent knows exactly what calls what and acts immediately.

CBM returned 72 callers across multiple hops. More breadth, no confidence scores. 967 tokens.

Shell gives you the most data. Synapse gives you the most precise data. CBM gives you the broadest data. But 140 lines of grep noise is how production breaks at 2am — the agent thinks it found all the callers, but half of them were comments.

"Can I delete this function?" — the query that separates the tools

This is where the three tools diverge completely.

Shell listed 50 defp declarations. It can find private functions. It cannot tell you if any of them are dead. 1,688 tokens of function names, zero analysis.

CBM returned 20 zero-caller function names via Cypher. 105 tokens. Compact — and that compactness is the problem. It's a list with no safety assessment, no confidence, and no distinction between "this is an entry point that's supposed to have zero internal callers" and "this is genuinely dead." The agent needs 5–10 follow-up queries to determine what's actually safe to delete.

Synapse returned a structured decision:

is_dead: false
safe_to_delete: false
confidence: low
deletion_risk: high
production_callers: 0
agent_instruction: "Do not delete source code
from this result; resolve the exact
symbol first."

526 tokens. One call. A complete answer — verdict, evidence, confidence, risk, and next step. No follow-ups.

This is the difference between a tool that gives you data and a tool that gives you a decision. One optimizes for tokens per query. The other optimizes for not causing a 2am incident.

"What breaks if I change this file?"

Shell: 313 lines mentioning "Store.ETS." 10,769 tokens. Every line that contains the string. No impact analysis. Just text matching.

CBM: 0 impacted items. Its change detection only works on git diffs — things that already changed. Ask it "what would break?" and it has nothing to say.

Synapse: 10 blast radius nodes, each with risk_level and risk_score. Coverage maps. Blind spots. 1,047 tokens. A structured impact assessment before you write a single line.

We're the only tool in this comparison that tells you what will break before you break it.

What it actually costs per month

Let's put the session numbers into dollars. At frontier model pricing ($3–$15 per million tokens), over a 20-day work month — based on full-session token counts from our security audit benchmark:

Synapse Pro at $19/month pays for itself in the first week — not from smaller payloads, but from fewer queries, fewer round trips, and fewer tokens wasted on false positives and follow-up exploration.

What only Synapse has

• Safe writes — simulates edits in memory, verifies every caller, auto-rolls back if anything would break. Neither shell nor CBM can write files at all.
• Dead code decisions — structured safe_to_delete verdict with confidence and evidence. CBM gives you a list of names; Synapse gives you a decision.
• Change review — blast radius with risk scores before you push. CBM only works post-hoc on git diffs.
• Stack trace resolution — paste a crash trace, get the root cause mapped to your code. No other tool does this.
• Knowledge cache — learns from searches and persists across sessions. The tool gets smarter the more you use it.
• Worktree overlay — sub-50ms delta indexing for parallel agents in Git worktrees.

What CBM does better (and I'm not going to pretend otherwise)

We lost the per-query token column. CBM also beats us in a few other places, and they're worth naming:

• 158 languages vs our 50+. We add languages on request — if yours isn't supported, open an issue and we'll add it.
• Cypher queries — a full query language for ad-hoc graph pattern matching. We give you structured tools. Different philosophies.
• 3D visualisation — a built-in UI. We had it. We took it out. It was expensive and nobody used it more than once.
• 43 agent auto-config surfaces on install. We're catching up.
• CLI mode — every CBM tool works as a one-shot shell command. Ours are MCP-only.

CBM is a genuinely good tool. If you need breadth and query language power, it's excellent. But it's read-only. It can tell you who calls a function. It can't stop you from breaking those callers.

Why we built this instead of using what existed

Every tool I tried — CBM, Cursor's built-in indexing, Greptile, Augment — could tell me what was in the codebase. None of them could tell me whether it was safe to change it.

That's the gap. That's why Synapse exists.

I didn't build another code explorer. I built a tool that prevents the thing every developer has experienced: an AI agent that confidently refactors a function, updates 2 callers, declares success, and silently breaks 3 others in sibling packages that it never found because grep doesn't understand module aliases.

Safe writes. Dead code decisions. Blast radius before you push. Stack trace resolution. Persistent learning. These aren't features on a comparison matrix. They're the difference between shipping with confidence and shipping with fingers crossed.

The free tier is the full product

I mean that literally. The free tier isn't a trial with a timer.

You get AST indexing across 50+ languages. Three search modes — semantic, symbol, regex. File inspection with outline compression. Standard file writes. SmartCrusher payload compression. OS-level background file watching. Cross-repo edge resolution. Git worktree overlay with sub-50ms delta indexing. Unlimited repositories. No time limits. No credit card.

Pro adds the safety layer — safe writes with auto-rollback, crash resolution, change review with blast radius, test intelligence, dead code decisions, and persistent knowledge cache. $19/month. Pays for itself in the first week.

curl -fsSL https://downloads.synapse-mcp.dev/install.sh | sh

All benchmark scripts and raw outputs are published for reproducibility. Run it yourself. Prove us wrong.