requests
@ 0e322af877· 14 queries44.9× fewer tokens| Backend | Recall | Found | Tokens/query | MRR |
|---|---|---|---|---|
| full-file | 1.00 | 100% | 41,729 | 1.00 |
| ripgrep | 0.79 | 71% | 26,543 | 0.60 |
| embedding-rag | 1.00 | 100% | 607 | 0.96 |
| neuralmind | 0.96 | 93% | 930 | 0.93 |
Reproducible4 pinned repos · 40 pre-registered queries
The public benchmark
A no-cherry-picking comparison of how much context different approaches put in an agent’s window to answer a real code question — and whether the objectively correct file actually makes it in. Cost and correctness are reported together, because a token-reduction number with no correctness number attached is meaningless.
Every query was registered before tuning. Every miss is published below. If you would rather not take any of it on trust, the whole thing reruns from a fresh clone with one command.
Rerun it yourself
git clone https://github.com/dfrostar/neuralmind && cd neuralmind pip install -e . tiktoken # source checkout ships the harness python -m evals.public.run # clones the pinned repos, prints the table
The harness lives in the source tree, not the PyPI wheel, so run it from a clone. The repos are cloned at fixed commit SHAs and the run is deterministic, so your numbers match the tables below to the token. Raw per-query data is committed in bench/public/results.json; the query set is in evals/public/manifest.json.
What it comes to
- Gold-file recall
93.75%
Weighted mean across 40 queries. Per repo it ranges 0.79–1.00 —
clickis the floor.- Tokens vs pasting files
45–257×
Fewer. Not uniform across repos, which is the honest picture rather than one cherry-picked ratio.
- Queries missed
4 of 40
Named individually below, with the files it retrieved instead.
How it is built to be hard to dismiss
Six design decisions, each aimed at a specific way a vendor benchmark usually fails.
- Real, recognizable repos
- requests, click, flask and rich, pinned to release SHAs. Anyone can check out the commit and audit. No vendor fixture, no repos picked for being easy.
- Objective gold, no LLM judge
- Each query's gold file is the definition site of a named symbol, verifiable with one ripgrep command. A backend answers if and only if the gold file lands in its assembled context. Nothing to rig.
- Cost and correctness together
- The headline is always recall at N× fewer tokens, never a lone ratio. A token-reduction number with no correctness number attached is meaningless.
- Strong baselines, disclosed
- Not just whole-file dumps. A keyword baseline (ripgrep) and a function-level vector RAG using the same encoder NeuralMind uses — which makes it, in effect, NeuralMind’s own retrieval core in isolation.
- Pre-registered queries
- Every query is committed to the manifest before tuning, and every one is reported. Losses are shown, not dropped.
- Deterministic
- Synapse injection is off, the repos are pinned, and re-running yields identical numbers. Your run matches this table to the token.
Results, per repo
Tokenizer: tiktoken o200k_base. Token counts are shown for every backend rather than a ratio, so the columns compare directly — including where NeuralMind is the more expensive one.
click
@ 874ca2bc1c· 7 queries99.6× fewer tokensThe weakest repo in the corpus — and the one where a bare vector index beats NeuralMind outright on recall.
| Backend | Recall | Found | Tokens/query | MRR |
|---|---|---|---|---|
| full-file | 1.00 | 100% | 78,514 | 1.00 |
| ripgrep | 0.79 | 71% | 45,059 | 0.60 |
| embedding-rag | 1.00 | 100% | 634 | 0.67 |
| neuralmind | 0.79 | 71% | 788 | 0.52 |
flask
@ c12a5d874c· 10 queries76.4× fewer tokens| Backend | Recall | Found | Tokens/query | MRR |
|---|---|---|---|---|
| full-file | 1.00 | 100% | 59,013 | 1.00 |
| ripgrep | 0.85 | 80% | 26,891 | 0.65 |
| embedding-rag | 0.95 | 90% | 687 | 0.73 |
| neuralmind | 0.95 | 90% | 772 | 0.78 |
rich
@ 7f580bdc70· 9 queries256.8× fewer tokens| Backend | Recall | Found | Tokens/query | MRR |
|---|---|---|---|---|
| full-file | 1.00 | 100% | 232,483 | 1.00 |
| ripgrep | 1.00 | 100% | 43,437 | 0.75 |
| embedding-rag | 1.00 | 100% | 677 | 0.94 |
| neuralmind | 1.00 | 100% | 905 | 0.80 |
Where NeuralMind loses
Two loss modes, reported plainly. Most vendor benchmarks do not have this section; it is here because a benchmark that never reports a loss is not evidence, it is advertising.
1 — The four gold-file misses
| Query | Gold file(s) | Retrieved instead |
|---|---|---|
| xfile-status-codesrequests | models.py, status_codes.py | models.py |
| echo-utilclick | utils.py | termui.py, _termui_impl.py, core.py |
| xfile-command-helpclick | core.py, formatting.py | decorators.py, core.py |
| xfile-dispatch-contextflask | app.py, ctx.py | app.py, views.py |
Three of the four are cross-file queries where the answer spans two modules and only one was retrieved. click accounts for two of them and is the weakest repo in the corpus at 0.79 recall; rich has none.
2 — Token cost against a bare vector index
On three of four repos the embedding-rag baseline matches or beats NeuralMind’s recall at fewer tokens, and on click it beats it outright — 1.00 against 0.79. Two things are true about that at once, and both belong on the page. That baseline is NeuralMind’s own encoder doing function-level retrieval, so the gap measures what the progressive-disclosure layer costs on top of raw top-k. And its cheaper number buys bare chunks, where NeuralMind spends the difference assembling a structured context — project map, signatures, call edges — meant for answering rather than locating. Gold-file recall measures locating. If locating a file is all you need, a bare vector index is cheaper, and we would rather say so here than have you find out later.
What this benchmark does not measure
Both omissions are deliberate, and both are measured elsewhere.
The synapse learning layer
Synapse weights learn from how you actually work, so they are session- and usage-dependent and cannot be part of a fixed, reproducible public number. Injection is off for every run above. The learning lift is measured separately by the synapse A/B eval, where CI asserts the direction — recall on is never worse than recall off, at a neutral token budget — rather than a magnitude.
End-to-end answer quality
Gold-file recall is deliberately a findability metric: objective, no judge, nothing to argue with. There is an opt-in answerability arm that grades real answers built from each backend’s actual window, with every transcript committed — but it is LLM-judged, so it stays a clearly labelled secondary signal and never the headline.
Run it on your own repo
These four repos are public so the numbers can be audited. The number that decides anything for you is the one from your codebase — neuralmind benchmark . prints it in about fifteen minutes.