# NeuralMind — the honest public benchmark

**What this is:** a reproducible, no-cherry-picking comparison of how much
context different approaches put in an agent's window to answer a real code
question, and whether the **objectively-correct file** actually makes it in.
Cost (tokens) and correctness (gold-file recall) are reported **together** — a
token-reduction number with no correctness number attached is meaningless.

**Reproduce it yourself (no trust required):**

```bash
git clone https://github.com/dfrostar/neuralmind && cd neuralmind
pip install -e . tiktoken           # source checkout — ships the evals/public harness
python -m evals.public.run          # clones the pinned repos, prints the table
# or, from the clone: neuralmind benchmark --public
```

> The benchmark harness (`evals/public`) ships in the **source tree**, not the
> PyPI wheel, so run it from a clone — `pip install neuralmind` alone won't have
> it. The pinned repos are cloned at fixed commit SHAs and the run is
> deterministic, so your numbers match the table below to the token.

---

## Method (designed to be hard to dismiss)

| Decision | Why |
|---|---|
| **Real, recognizable repos** — `requests`, `click`, `flask`, `rich`, pinned to release SHAs | Anyone can `git checkout <sha>` and audit. No vendor fixture. Household-name repos, not cherry-picked easy ones. |
| **Objective gold, no LLM judge** | Each query's gold file is the **definition site** of a named symbol, verifiable with one `rg` command. A baseline "answers" iff the gold file lands in its assembled context. Deterministic, nothing to rig. |
| **Cost + correctness reported jointly** | The headline is "recall at N× fewer tokens," never a lone ratio. |
| **Strong baselines, disclosed** | Not just naive whole-file dumps — we include keyword (`ripgrep`) and a function-level vector RAG using the *same encoder* NeuralMind uses. |
| **Pre-registered queries, every one reported** | Queries are committed in `evals/public/manifest.json` before tuning; losses are shown, not hidden. |
| **Deterministic** | Synapse injection is OFF (see "What this does *not* measure"). Re-running yields identical numbers. |

### The baselines

- **`full-file`** — paste every source file into context. Recall is trivially
  1.0; cost is the whole repo. This is the honest ceiling the "just give the
  model the files" workflow actually pays.
- **`ripgrep`** — extract query keywords, rank files by match count, open the
  top-5 whole files (what "just grep and read the files" costs).
- **`embedding-rag`** — top-8 function/class chunks retrieved by the same encoder
  NeuralMind uses, sending only the retrieved chunks. A strong, cheap vector-RAG
  baseline. **Note:** this is effectively *NeuralMind's own vector-retrieval core
  in isolation*, so the `neuralmind` vs `embedding-rag` gap isolates what the
  progressive-disclosure assembly layer adds (and costs) on top of raw top-k.
- **`neuralmind`** — `NeuralMind.query`: progressive L0–L3 disclosure assembling
  a compact, structured context (project map + relevant symbols + call edges).

Scoring reuses `neuralmind/quality.py` verbatim — the same metric code the CI
quality gate runs — so these are not numbers invented for a press release.

---

## Results

_Tokenizer: tiktoken `o200k_base`. Generated by `python -m evals.public.run`;
raw per-query data in [`bench/public/results.json`](../../bench/public/results.json)._

### `requests` @ `0e322af877` — 14 pre-registered queries

| backend | gold-file recall | found-rate | mean tokens/query | vs full-file |
|---|---:|---:|---:|---:|
| `full-file` | 1.00 | 100% | 41,729 | 1× |
| `ripgrep` | 0.79 | 71% | 26,543 | 1.6× |
| `embedding-rag` | 1.00 | 100% | 607 | 69× |
| **`neuralmind`** | **1.00** | **100%** | **1,095** | **38×** |

### `click` @ `874ca2bc1c` — 7 pre-registered queries

| backend | gold-file recall | found-rate | mean tokens/query | vs full-file |
|---|---:|---:|---:|---:|
| `full-file` | 1.00 | 100% | 78,514 | 1× |
| `ripgrep` | 0.79 | 71% | 45,059 | 1.7× |
| `embedding-rag` | 1.00 | 100% | 649 | 121× |
| **`neuralmind`** | **1.00** | **100%** | **924** | **85×** |

---

## What the numbers honestly say

1. **Against what developers actually do today — paste files or grep — NeuralMind
   is a large, real win.** It reaches **100% gold-file recall at 38–85× fewer
   tokens** than pasting the files, and it beats `ripgrep` on *both* axes (full
   recall vs grep's 0.79, at fewer tokens). Most agents don't have a tuned
   function-level vector index sitting there; they read files or grep. That is
   the baseline NeuralMind replaces.

2. **A well-tuned vector RAG is also excellent at *findability* — and we show it.**
   On these repos `embedding-rag` matches NeuralMind's recall at fewer tokens.
   We do not hide this. Two honest caveats: (a) that baseline *is* NeuralMind's
   own encoder doing function-level retrieval, and (b) its "cost" is the bare
   retrieved chunks — NeuralMind spends its extra tokens assembling a *structured,
   readable* context (project map, signatures, call edges) an agent uses to
   **answer**, not just to locate the file. Pure gold-file recall measures
   locating, not answering.

3. **`ripgrep` is the cautionary tale.** Cheap-ish, but it *misses the right file
   29% of the time* (recall 0.79) — keyword search has no notion of meaning.

### Where NeuralMind loses

On both repos NeuralMind missed **zero** gold files (found-rate 100%). The one
axis where it is *not* the leader is **token cost vs. a bare top-k vector
retrieval** — it spends ~1.5–2× the tokens of `embedding-rag` to deliver
assembled context at the same recall. We report that plainly; if your only need
is "which file," a bare vector index is cheaper.

## What this benchmark does *not* measure (on purpose)

- **The synapse learning layer.** Synapse weights are session- and
  usage-dependent — they learn from how *you* work — so they cannot be part of a
  *fixed, reproducible* public number. Injection is OFF here. The learning lift
  is measured separately and reproducibly by the **synapse A/B eval**
  (`tests/benchmark/run.py`, Phase 2): **+11.7 points** top-k hit-rate, budget-
  neutral, on the reference fixture. That is the differentiator a static index
  structurally cannot copy.
- **End-to-end answer quality.** Gold-file recall is deliberately a *findability*
  metric (objective, no judge). The opt-in **answerability arm** (`--judge`) adds
  the *answering* signal — see below — but it stays a clearly-labeled secondary,
  never the headline.

## Answerability arm — `--judge` (opt-in, secondary signal)

Gold-file recall measures *locating* the right file, not *answering* the
question. The opt-in answerability arm closes that gap honestly:

```bash
ANTHROPIC_API_KEY=… python -m evals.public.run --judge   # off by default
```

For each query it takes **the real context each backend would put in the
window** — whole files for `full-file`/`ripgrep`, retrieved chunks for
`embedding-rag`, the compact L0–L3 assembly for `neuralmind` — and asks a pinned
model (`claude-opus-4-8`) to answer **using only that context** (it must say
"insufficient context" when the window can't support an answer). A separate judge
call grades each answer against the **same def-site gold anchor** (the
`oracle_symbol`) on a 0–2 scale, plus a `grounded` flag.

**Why it's hard to dismiss:**

- **Same prompts, same pinned model, every backend** — each is judged on its own
  real window, so a low-recall window scores low instead of being papered over
  from the model's prior knowledge.
- **Published provenance** — the answerer prompt, the judge rubric, the pinned
  model id, and **every raw transcript** (question, context tokens, answer,
  verdict, rationale) are committed under
  [`bench/public/judge/`](../../bench/public/judge/). Re-score it, or swap the
  model, yourself.
- **Off the deterministic path** — needs `ANTHROPIC_API_KEY`, never runs in CI,
  and the recall table above is byte-identical with or without `--judge`.

**Honest caveats:** it's LLM-judged (not deterministic like recall); a single
judge model is one viewpoint (disclosed, transcripts committed so anyone can
re-score with another); and it's a *secondary* signal — the recall-at-N×-tokens
headline stays primary.

## Extending the corpus

The corpus spans **four pinned repos / 40 pre-registered queries**: `requests`
(14) and `click` (7) — the original headline pair, with committed numbers above
— plus **`flask` @ `c12a5d87` (10)** and **`rich` @ `7f580bdc` (9)**, added to
harden the "you picked easy repos" critique with two more household-name
projects. Every `flask`/`rich` query's gold file is the **objective definition
site** of its named symbol, verified with `rg` against the pinned commit (e.g.
`class Flask` in `app.py`, `class Console` in `console.py`) — pre-registered in
`evals/public/manifest.json` *before* any run, exactly as the methodology
requires (gold first, numbers second). Their cost+correctness rows regenerate
with `python -m evals.public.run` (the committed `requests`/`click` snapshot
stays a subset of the manifest, so nothing is fabricated in between).

Add a repo the same way: append to `evals/public/manifest.json` (pin the commit,
give each query an objective def-site gold file) and re-run. Community-contributed
real-repo numbers go through the existing `neuralmind benchmark . --contribute`
path.

## Competitor head-to-head — `codebase-memory-mcp` 0.8.1

The obvious incumbent (DeusData, single-binary, on-device embeddings, **no LLM
API key**) runs headless, so we ran a real, reproducible row on the **same**
pinned repos, same questions, same objective def-site gold, same `quality.py`
scorer, retrieval depth matched to `embedding-rag` (top-8). Raw per-query traces:
[`bench/public/competitor/`](../../bench/public/competitor/). Reproduce:
`python -m evals.public.competitor`.

| repo | backend | gold-file recall | found-rate | MRR (rank quality) | mean tokens |
|---|---|---:|---:|---:|---:|
| requests | `codebase-memory-mcp` | 0.50 | 43% | 0.23 | 25,214 |
| requests | **neuralmind** | **1.00** | **100%** | **0.96** | **1,095** |
| click | `codebase-memory-mcp` | 0.64 | 57% | 0.50 | 38,538 |
| click | **neuralmind** | **1.00** | **100%** | **0.60** | **924** |

**At matched retrieval depth, NeuralMind finds the objectively-correct file every
time and ranks it far higher** (MRR 0.96 vs 0.23 on requests), while the
competitor surfaces the gold file in its top-8 only ~half the time — and the
files it does surface cost an order of magnitude more tokens to read.

### Fairness & honest caveats (don't skip)

- **Most-favorable keyword mapping.** The competitor takes a keyword array, not
  free text. We tested three reproducible mappings and used the one **best for the
  competitor** (all question words) — so this can't be dismissed as crippling it.
- **Pure retrieval on both sides, no agent loop.** This measures retrieval
  ranking given a question — exactly how we test NeuralMind's own `search`. The
  competitor's *published* numbers (~90% of an "Explorer" agent across 31 scored
  languages; C at **0.58**) come from an **LLM-driven** agent loop that isn't
  reproducible without paying for an LLM; we don't claim to reproduce that, and we
  cite their figures as-is. Their "158 languages" is vendored grammars, not all
  benchmarked.
- **Cost is a proxy** (tokens of the whole files the competitor surfaces at depth
  8, since it returns paths the agent then reads — analogous to `ripgrep`).
- Full provenance + pinned version + exact commands:
  [`bench/public/competitor/REPRODUCE.md`](../../bench/public/competitor/REPRODUCE.md).