The public benchmark

Rerun it yourself

git clone https://github.com/dfrostar/neuralmind && cd neuralmind
pip install -e . tiktoken                            # source checkout ships the harness
NEURALMIND_ORT_THREADS=1 python -m evals.public.run  # clones the pinned repos, prints the table

The harness lives in the source tree, not the PyPI wheel, so run it from a clone. The repos are cloned at fixed commit SHAs and nothing in the run is random, so recall and found-rate should match the tables below exactly. Token counts can shift slightly between machines — CI runners without AVX-512 land up to 0.8% off on a per-repo mean; NEURALMIND_ORT_THREADS=1 matches CI’s setup (what moves, and the evidence). Raw per-query data is committed in bench/public/results.json; the query set is in evals/public/manifest.json.

What it comes to

Two kinds of result, side by side: retrieval, where NeuralMind gets the right code into the window in far fewer tokens, and task outcomes, where a pre-registered pilot found it made no difference.

How it is built to be hard to dismiss

Six design decisions, each aimed at a specific way a vendor benchmark usually fails.

Real, recognizable repos

requests, click, flask and rich, pinned to release SHAs. Anyone can check out the commit and audit. No vendor fixture, no repos picked for being easy.

Objective gold, no LLM judge

Each query's gold file is the definition site of a named symbol, verifiable with one ripgrep command. A backend answers if and only if the gold file lands in its assembled context. Nothing to rig.

Cost and correctness together

The headline is always recall at N× fewer tokens, never a lone ratio. A token-reduction number with no correctness number attached is meaningless.

Strong baselines, disclosed

Not just whole-file dumps. A keyword baseline (ripgrep) and a vector baseline that returns the top 8 entries of NeuralMind’s own vector index — in effect, NeuralMind’s retrieval core in isolation. It sends a symbol’s name, module and docstring, not its code, so its token count is a floor: a chunk RAG that sends code bodies would pay more.

Pre-registered queries

Every query is committed to the manifest before tuning, and every one is reported. Losses are shown, not dropped.

Deterministic per machine

Synapse injection is off, the repos are pinned and nothing is sampled, so a re-run on the same machine is byte-identical. Across machines, recall and found-rate have come back identical. On the GitHub-hosted CI runners without AVX-512, token counts shift slightly, by up to 0.8% on a per-repo mean in this run, and so does the vector-RAG baseline’s MRR on click; an Apple M3, also without AVX-512, matched the committed run exactly.

Results, per repo

Tokenizer: tiktoken o200k_base. Token counts are shown for every backend rather than a ratio, so the columns compare directly, including where NeuralMind is the more expensive one.

What this does not measure