2ms vs. 12 Seconds: Benchmark Proves Why AI Agents Need AST Graphs, Not Grep Chains

We spend a lot of time watching AI coding agents work. The pattern is always the same: grep, read a file, grep again, read another file, and before you know it the agent has burned a dozen tool calls just to figure out where something lives in the codebase. We wanted to know exactly how much time that wastes, so we ran a benchmark.

The task was real, not synthetic. We asked an agent to verify a multi-agent instruction-writing feature that's spread across four separate sub-modules. It had to:

• Check whether multi-agent instruction writing is wired into the CLI installer.
• Verify the supported editor harnesses in the configuration module.
• Inspect the initialize JSON-RPC handler in the protocol layer.
• Confirm the dynamic readiness instruction generation in the tool catalogue.

The kind of thing you'd ask an agent to do ten times a day. Nothing exotic. We ran it two ways — the old grep-and-read loop that every agent does by default, and a single Synapse graph query — and logged everything.

What the grep loop actually looks like

Without an AST graph, the agent has to guess at file paths and crawl the filesystem with text matching. Here's the full log of every tool call it made:

Step 1: list_dir("/path/to/project") → Returns 63 files
Step 2: grep_search("ruleTargets") → Scans filesystem... (1,200 ms)
Step 3: view_file("install.js", 1-60) → Partial slice (800 ms)
Step 4: view_file("install.js", 270-340) → Locates rule targets (900 ms)
Step 5: view_file("mcp-config.js", 1-50) → Reads headers (700 ms)
Step 6: view_file("mcp-config.js", 180-260) → Reads editor list (950 ms)
Step 7: view_file("rpc-handler.ex", 50-89) → Inspects initialize method (1,100 ms)
Step 8: view_file("tool_catalogue.ex", 1-60) → Reads catalogue (850 ms)
Step 9: view_file("tool_catalogue.ex", 1300-1389) → Reads instructions (1,400 ms)

Nine sequential tool calls. 12.4 seconds of wall time. 8,400 tokens of raw source code pulled into context — and most of it was noise. Steps 3 and 5 read file headers the agent didn't need. Steps 4 and 6 were second guesses because the first slice didn't contain what the agent was looking for. This is the part that doesn't show up in the latency number: the agent is flying blind, slicing files by line number, hoping the relevant code is in the range it picked.

What one graph query looks like

Then we pointed the same agent at Synapse and asked it to do the same task with a single synapse_get_context call against the in-memory AST graph:

call_mcp_tool(ServerName: "synapse", ToolName: "synapse_get_context", Arguments: {
  action: "find",
  query: "multi-agent instruction writing installer mcp-config rpc tool_catalogue",
  repo_id: "my-project"
})

The graph already knows where every function lives, who calls it, and what module it belongs to. So instead of guessing, the query resolves to the exact chunks across all four modules in one pass. Here's the response:

{
  "_synapse": {
    "boot": "ready",
    "ready": true,
    "savings": {
      "calls_saved": 9,
      "tokens_saved": 2136
    }
  },
  "count": 7,
  "format": "markdown",
  "results": [ ... Full AST chunk boundaries, caller/callee graphs, and exact line definitions ... ]
}

One call. 2 milliseconds. 3,660 tokens — and those tokens are the actual AST chunks with caller/callee edges, not raw file slices. The payload goes through SmartCrusher before it hits the wire, which relativises paths and shortens structural keys, so what the agent receives is already compressed down to the structural minimum.

The numbers, side by side

The number that actually matters

The 6,200× latency number is the flashy one, and it's real, but it's not the most useful number on that table. The latency gap shrinks the moment you're doing anything more complex than a local lookup — the agent still has to think about what it found. The numbers that matter to us are the call count and the token count.

Nine calls down to one means the agent isn't context-switching between grepping, reading, grepping again, and re-reading. Every round trip is a chance to lose the thread, pick the wrong file, or burn tokens re-reading something it already saw. 8,400 tokens down to 3,660 means the context window has room for the actual work instead of being half-full of file slices the agent is done with. And getting the full caller/callee graph instead of line ranges means the agent isn't guessing at dependencies — it knows them.

This is what we built Synapse for. Not to win a latency benchmark, but to stop agents from spending 80% of their budget on file I/O and re-reading code they've already seen. The graph is local, in ETS, no network. The query is a graph traversal, not a filesystem scan. The response is compressed to the structural minimum. We ran this benchmark because we wanted to know if it actually made a difference. It did.