neuralmind

NeuralMind — the honest public benchmark

What this is: a reproducible, no-cherry-picking comparison of how much context different approaches put in an agent’s window to answer a real code question, and whether the objectively-correct file actually makes it in. Cost (tokens) and correctness (gold-file recall) are reported together — a token-reduction number with no correctness number attached is meaningless.

Reproduce it yourself (no trust required):

git clone https://github.com/dfrostar/neuralmind && cd neuralmind
pip install -e . tiktoken           # source checkout — ships the evals/public harness
python -m evals.public.run          # clones the pinned repos, prints the table
# or, from the clone: neuralmind benchmark --public

The benchmark harness (evals/public) ships in the source tree, not the PyPI wheel, so run it from a clone — pip install neuralmind alone won’t have it. The pinned repos are cloned at fixed commit SHAs and the run is deterministic, so your numbers match the table below to the token.

---

Method (designed to be hard to dismiss)

Decision	Why
Real, recognizable repos — requests, click, flask, rich, pinned to release SHAs	Anyone can git checkout <sha> and audit. No vendor fixture. Household-name repos, not cherry-picked easy ones.
Objective gold, no LLM judge	Each query’s gold file is the definition site of a named symbol, verifiable with one rg command. A baseline “answers” iff the gold file lands in its assembled context. Deterministic, nothing to rig.
Cost + correctness reported jointly	The headline is “recall at N× fewer tokens,” never a lone ratio.
Strong baselines, disclosed	Not just naive whole-file dumps — we include keyword (ripgrep) and a function-level vector RAG using the same encoder NeuralMind uses.
Pre-registered queries, every one reported	Queries are committed in evals/public/manifest.json before tuning; losses are shown, not hidden.
Deterministic	Synapse injection is OFF (see “What this does not measure”). Re-running yields identical numbers.

The baselines

• full-file — paste every source file into context. Recall is trivially 1.0; cost is the whole repo. This is the honest ceiling the “just give the model the files” workflow actually pays.
• ripgrep — extract query keywords, rank files by match count, open the top-5 whole files (what “just grep and read the files” costs).
• embedding-rag — top-8 function/class chunks retrieved by the same encoder NeuralMind uses, sending only the retrieved chunks. A strong, cheap vector-RAG baseline. Note: this is effectively NeuralMind’s own vector-retrieval core in isolation, so the neuralmind vs embedding-rag gap isolates what the progressive-disclosure assembly layer adds (and costs) on top of raw top-k.
• neuralmind — NeuralMind.query: progressive L0–L3 disclosure assembling a compact, structured context (project map + relevant symbols + call edges).

Scoring reuses neuralmind/quality.py verbatim — the same metric code the CI quality gate runs — so these are not numbers invented for a press release.

---

Results

Tokenizer: tiktoken o200k_base. Generated by python -m evals.public.run; raw per-query data in bench/public/results.json.

requests @ 0e322af877 — 14 pre-registered queries

backend	gold-file recall	found-rate	mean tokens/query	vs full-file
full-file	1.00	100%	41,729	1×
ripgrep	0.79	71%	26,543	1.6×
embedding-rag	1.00	100%	607	69×
neuralmind	1.00	100%	1,095	38×

click @ 874ca2bc1c — 7 pre-registered queries

backend	gold-file recall	found-rate	mean tokens/query	vs full-file
full-file	1.00	100%	78,514	1×
ripgrep	0.79	71%	45,059	1.7×
embedding-rag	1.00	100%	649	121×
neuralmind	1.00	100%	924	85×

---

What the numbers honestly say

1. Against what developers actually do today — paste files or grep — NeuralMind is a large, real win. It reaches 100% gold-file recall at 38–85× fewer tokens than pasting the files, and it beats ripgrep on both axes (full recall vs grep’s 0.79, at fewer tokens). Most agents don’t have a tuned function-level vector index sitting there; they read files or grep. That is the baseline NeuralMind replaces.

2. A well-tuned vector RAG is also excellent at findability — and we show it. On these repos embedding-rag matches NeuralMind’s recall at fewer tokens. We do not hide this. Two honest caveats: (a) that baseline is NeuralMind’s own encoder doing function-level retrieval, and (b) its “cost” is the bare retrieved chunks — NeuralMind spends its extra tokens assembling a structured, readable context (project map, signatures, call edges) an agent uses to answer, not just to locate the file. Pure gold-file recall measures locating, not answering.

3. ripgrep is the cautionary tale. Cheap-ish, but it misses the right file 29% of the time (recall 0.79) — keyword search has no notion of meaning.

Where NeuralMind loses

On both repos NeuralMind missed zero gold files (found-rate 100%). The one axis where it is not the leader is token cost vs. a bare top-k vector retrieval — it spends ~1.5–2× the tokens of embedding-rag to deliver assembled context at the same recall. We report that plainly; if your only need is “which file,” a bare vector index is cheaper.

What this benchmark does not measure (on purpose)

• The synapse learning layer. Synapse weights are session- and usage-dependent — they learn from how you work — so they cannot be part of a fixed, reproducible public number. Injection is OFF here. The learning lift is measured separately and reproducibly by the synapse A/B eval (tests/benchmark/run.py, Phase 2): +11.7 points top-k hit-rate, budget- neutral, on the reference fixture. That is the differentiator a static index structurally cannot copy.
• End-to-end answer quality. Gold-file recall is deliberately a findability metric (objective, no judge). The opt-in answerability arm (--judge) adds the answering signal — see below — but it stays a clearly-labeled secondary, never the headline.

Answerability arm — --judge (opt-in, secondary signal)

Gold-file recall measures locating the right file, not answering the question. The opt-in answerability arm closes that gap honestly:

ANTHROPIC_API_KEY=… python -m evals.public.run --judge   # off by default

For each query it takes the real context each backend would put in the window — whole files for full-file/ripgrep, retrieved chunks for embedding-rag, the compact L0–L3 assembly for neuralmind — and asks a pinned model (claude-opus-4-8) to answer using only that context (it must say “insufficient context” when the window can’t support an answer). A separate judge call grades each answer against the same def-site gold anchor (the oracle_symbol) on a 0–2 scale, plus a grounded flag.

Why it’s hard to dismiss:

• Same prompts, same pinned model, every backend — each is judged on its own real window, so a low-recall window scores low instead of being papered over from the model’s prior knowledge.
• Published provenance — the answerer prompt, the judge rubric, the pinned model id, and every raw transcript (question, context tokens, answer, verdict, rationale) are committed under bench/public/judge/. Re-score it, or swap the model, yourself.
• Off the deterministic path — needs ANTHROPIC_API_KEY, never runs in CI, and the recall table above is byte-identical with or without --judge.

Honest caveats: it’s LLM-judged (not deterministic like recall); a single judge model is one viewpoint (disclosed, transcripts committed so anyone can re-score with another); and it’s a secondary signal — the recall-at-N×-tokens headline stays primary.

Extending the corpus

The corpus spans four pinned repos / 40 pre-registered queries: requests (14) and click (7) — the original headline pair, with committed numbers above — plus flask @ c12a5d87 (10) and rich @ 7f580bdc (9), added to harden the “you picked easy repos” critique with two more household-name projects. Every flask/rich query’s gold file is the objective definition site of its named symbol, verified with rg against the pinned commit (e.g. class Flask in app.py, class Console in console.py) — pre-registered in evals/public/manifest.json before any run, exactly as the methodology requires (gold first, numbers second). Their cost+correctness rows regenerate with python -m evals.public.run (the committed requests/click snapshot stays a subset of the manifest, so nothing is fabricated in between).

Add a repo the same way: append to evals/public/manifest.json (pin the commit, give each query an objective def-site gold file) and re-run. Community-contributed real-repo numbers go through the existing neuralmind benchmark . --contribute path.

Competitor head-to-head — codebase-memory-mcp 0.8.1

The obvious incumbent (DeusData, single-binary, on-device embeddings, no LLM API key) runs headless, so we ran a real, reproducible row on the same pinned repos, same questions, same objective def-site gold, same quality.py scorer, retrieval depth matched to embedding-rag (top-8). Raw per-query traces: bench/public/competitor/. Reproduce: python -m evals.public.competitor.

repo	backend	gold-file recall	found-rate	MRR (rank quality)	mean tokens
requests	codebase-memory-mcp	0.50	43%	0.23	25,214
requests	neuralmind	1.00	100%	0.96	1,095
click	codebase-memory-mcp	0.64	57%	0.50	38,538
click	neuralmind	1.00	100%	0.60	924

At matched retrieval depth, NeuralMind finds the objectively-correct file every time and ranks it far higher (MRR 0.96 vs 0.23 on requests), while the competitor surfaces the gold file in its top-8 only ~half the time — and the files it does surface cost an order of magnitude more tokens to read.

Fairness & honest caveats (don’t skip)

• Most-favorable keyword mapping. The competitor takes a keyword array, not free text. We tested three reproducible mappings and used the one best for the competitor (all question words) — so this can’t be dismissed as crippling it.
• Pure retrieval on both sides, no agent loop. This measures retrieval ranking given a question — exactly how we test NeuralMind’s own search. The competitor’s published numbers (~90% of an “Explorer” agent across 31 scored languages; C at 0.58) come from an LLM-driven agent loop that isn’t reproducible without paying for an LLM; we don’t claim to reproduce that, and we cite their figures as-is. Their “158 languages” is vendored grammars, not all benchmarked.
• Cost is a proxy (tokens of the whole files the competitor surfaces at depth 8, since it returns paths the agent then reads — analogous to ripgrep).
• Full provenance + pinned version + exact commands: bench/public/competitor/REPRODUCE.md.

This site is open source. Improve this page.