Measuring AI agent memory across a major refactor
A ~9,300-node TypeScript SaaS platform was rebuilt with NeuralMind watching. The memory didn't go stale — it grew into the new architecture. Here are the numbers, where each one came from, and how to run the same measurement on your repo.
Avg token reduction
48.8×
~1,033 tokens/query vs the pre-v4.5.0 fixed 50K-token estimate
Personal synapse edges
36 → 135
the learning layer tracked the new code
Full rebuild
326 s
then back to ~30 s incremental
The setting
A private, mid-size TypeScript SaaS platform (~9,300 indexed nodes), maintained by NeuralMind's own maintainer — see the honesty notes below. A major internal rebuild ("Phase 3 → Phase 4") landed a new backend module, a shared business-logic layer, an admin UI, and end-to-end specs.
NeuralMind ran throughout with lifecycle hooks installed, so the synapse layer observed the work as it happened. The repo is private, so the numbers are anonymized; every one of them came from the shipped CLI.
The numbers
| Metric | Before (Phase 3) | After (Phase 4) | Change |
|---|---|---|---|
| Total nodes | 9,190 | 9,293 | +103 |
| Communities | — | 810 | new resolution — no comparable before |
| Personal synapse edges | 36 | 135 | +275% |
| Shared synapse edges | 10,079 | 10,183 | +104 |
| Shared edge weight | 2,774.73 | 2,924.66 | +5.4% |
| Wake-up tokens | — | 455 | |
| Avg query tokens | — | 1,033 | |
| Avg token reduction | — | 48.8× | vs the fixed 50K-token estimate the CLI used before v4.5.0 |
| Full --force rebuild | — | 326 s (~5.4 min) | incremental rebuilds ~30 s after |
What the numbers say
48.8× token reduction. After the rebuild, an average code question cost ~1,033 tokens of context, against the 50,000-token estimate the CLI then used for what a naive "load the relevant files" approach spends. That is one repo's ratio from neuralmind benchmark ., consistent with the 12–50× field-report range, measured the same way (by neuralmind benchmark before v4.5.0, against its fixed 50K-token estimate) — not a prediction for your repo.
Personal synapse edges tripled (36 → 135). The Hebbian synapse layer learned co-activations across the new code as it was being written and queried. The memory didn't go stale through the rebuild; it grew into the new architecture.
Shared edge weight +5.4%. The static graph's cross-links got denser as the new shared business-logic layer connected previously separate areas — densification, not just growth: edge weight rose 5.4% on only 1.1% more nodes.
Rebuild cost stayed flat. One full --force rebuild at 326 s to pick up the new structure, then incremental rebuilds back to ~30 s. Re-indexing is not a tax you pay per edit.
Where each number comes from
Honesty first: the before/after table is hand-assembled from the output of several commands — there is no single neuralmind compare-phases command.
| Metric | Command |
|---|---|
| Total nodes, communities | neuralmind stats . --json |
| Personal/shared edges, edge weight | neuralmind stats . — the Memory namespaces block |
| Wake-up / query tokens, reduction | neuralmind benchmark . |
| Rebuild time | timed neuralmind build . --force |
| Synapses fired per query | neuralmind metrics . |
| Real logged spend | neuralmind savings . |
Run the same measurement on your refactor
1. Snapshot before you start
neuralmind stats . --json > .neuralmind-before.json
neuralmind benchmark . # baseline reduction number
2. Refactor with the memory watching
The synapse layer only learns from what it observes — install hooks before the work starts.
neuralmind install-hooks .
neuralmind watch & # optional: always-on learning from edits
3. Rebuild and snapshot after
time neuralmind build . --force
neuralmind stats . --json > .neuralmind-after.json
4. Re-benchmark and price it
neuralmind benchmark . # reduction on the post-refactor graph
neuralmind savings . # what your logged queries cost vs naive
Diff the two JSON files. The interesting deltas: node count (did the graph track the new code?), personal edges (did the memory learn the new co-activations?), shared edge weight (did the graph get denser or just bigger?). The full step-by-step walkthrough lives in the docs.
Honesty notes
One repo, one developer, measured by the maintainer on a private codebase. It is an existence proof and a recipe, not an independent benchmark. You cannot re-run it on the source repo — but you can run the identical method on yours, which is the point.
48.8× is a reduction in retrieval input tokens, not your total LLM bill. Whether fewer retrieval tokens lower what a task costs was measured later, by a pre-registered 20-task pilot on a different repo: it found no difference.
The "naive" side of this ratio is an estimate. This report predates v4.5.0, when neuralmind benchmark . divided a fixed 50,000-token estimate of naive context by the measured tokens per question (~1,033 here); it did not count this repo's actual size. The measured half is the tokens per question. Since v4.5.0 the CLI divides the measured size of the code the index covers instead; this repo has not been re-run with it, so 48.8× stays the pre-v4.5.0 figure. The public benchmark measures against every source file too.
What this report can't show. That the synapse layer improves retrieval here: there is no production fitness data, and the personal-edge growth above is activity, not a measured gain. That team memory learns: structural seeds exist, but learning needs weeks of sessions with a co-view signal this repo didn't have. And no claim ofzero code egress: the agent still sends its context to its model. What NeuralMind itself does is transmit no repository content and send no telemetry, by default.
The CI-gated, reproducible numbers live in Benchmarks & Results and on the benchmark page; this page is deliberately labeled a field report.