I built the memory layer. Then the measurement came due.
Every serious agent program converges on the same memory shape: a store of decisions, conventions, and the reasons behind them that outlives any session; an ingest pipeline that keeps it current; retrieval that brings the right piece back mid-task. The promise is compounding, explain a rule once and no agent breaks it again. The costs are concrete and continuous, the store, the ingest, retrieval calls on every task. And the risk almost nobody prices is the moment of use: the agent pulls back the wrong thing at the wrong moment, and the work gets worse.
I have run that stack for months. Two earlier notes in this archive built it: one chose the memory substrate against an adoption bar, the next argued for owning the memory-and-policy layer above the swappable model. The layer holds the fleet's decisions and conventions, ingests on a cadence, and a deterministic gate forces agents to consult it before substantive edits. What none of that settles is the only question that matters in production: does the layer change what the agent does? Not whether ingest runs, not whether retrieval returns the right passage; whether the work comes out different. Retrieval benchmarks measure the store; nothing in them measures the moment of use, where the retrieved convention meets the model's own prior and one of them wins. So the measurement came due, pre-registered, with a grader that cannot be argued with. The result came first, above, because everything else in this note hangs off it.
One convention, four delivery channels, a grader that cannot be argued with.
The task is real repository work: extend a cross-boundary sharing rule by allowing a new namespace. The idiomatic fix is one line in a policy configuration file. The tempting wrong fix also hardcodes the namespace into code defaults, which works today and quietly breaks the policy contract tomorrow. The convention that separates them, configuration is policy, code carries primitives, is exactly the kind of institutional knowledge a memory layer exists to deliver. The experiment holds the convention constant and varies only the channel that delivers it.
Four arms, eight seeded runs each, the same frontier coding model throughout, thirty-two real sessions, plus a nine-run controlled follow-up. A wild arm, with the convention stripped from every channel. A harness-only arm, with full repository context but no memory product. An injected arm, which received the convention as one explicit, versioned sentence in the harness context. And an equipped arm, with the live retrieval layer connected and consultation forced: the agent queried memory about nine times per run, and the retrieval verifiably returned the relevant convention. The convention itself was scrubbed from code comments and documentation, so each arm could learn it only through its own channel. The grader is deterministic and separates three verdicts: functionally works; idiomatically right; and the verdict this experiment exists to catch, masked, meaning functionally works while quietly breaking the convention.
Same memory, two channels: eight of eight against one of eight.
Through the versioned-sentence channel, the convention held in every run, eight of eight. Through the retrieval channel, it held in one of eight, Fisher exact p = 0.0014 against the sentence, statistically indistinguishable from giving the agent nothing on that dimension, and directionally below the plain harness baseline. In the controlled follow-up that isolated the layer cleanly, three arms by three seeds, the retrieval layer's marginal value over an equivalent-discipline harness was zero on every metric.
Read the ordering, not just the winner. Full repository context alone produced the idiomatic fix half the time, so the harness was carrying real weight before any memory product entered. The versioned sentence turned half into certainty. And the retrieval layer, delivering the same convention through search instead of standing context, not only failed to add value on this task; it trended below the harness baseline. The pre-registered decision rule was written before the runs: if the equipped arm lands where the harness-only arm lands, the win is discipline and context, not memory recall. It fired. Note what this is not: it is not memory losing to no-memory. Both winning arms carried the institution's knowledge. The measurement separated delivery channels for the same knowledge, and the explicit, versioned channel crushed the search-and-inject channel.
Consulted, returned, ignored. And the field has located the same broken half.
The transcripts show the failure precisely. Retrieval worked: the convention came back, run after run. But retrieval also dumped the repository's existing hardcoded defaults into the context window, and the model reasoned from what it saw: add the namespace to the configuration and to the code defaults, belt and suspenders. Functionally green, convention broken, masked. That is the context-distraction failure mode of retrieval-augmented systems, measured end to end on a production-shaped task rather than a QA benchmark. The second mechanism is quieter: my agent consulted memory nine times per run only because a deterministic gate forced it to; in earlier unforced runs it never queried the layer spontaneously at all. Availability is not use, and retrieval is not adherence, which matches what the faithfulness literature now reports at benchmark scale: agents disregard or misread the condensed experience they wrote for themselves.
The frontier arrived at the same diagnosis from the other direction, weeks after my runs. AutoMem treats memory management, what to write, what to condense, what to fetch, when to trust it against a prior, as a trainable cognitive skill, and on three long-horizon game environments the full loop multiplies task performance by two to four times, bringing a 32-billion-parameter open model to the level of frontier systems on those benchmarks. The detail that matters here, computed from the paper's own ablation table (Table 1): eighty to eighty-nine percent of the total lift comes from the structure half, scaffold code, prompts, and file schemas around the model, which involves no weight training at all. The reading for a production team is not that anyone should train weights. It is that the broken half of agent memory is the same half my transcripts caught failing: not storage, not retrieval, but the management and use of what comes back. Those magnitudes live on games and personalization benchmarks and are subsidized by frontier models in the outer loop, so treat them as direction, not promise.
The channel that won, specified, because it is the whole point.
Call the finding what it is. What beat my retrieval layer was not the absence of memory; it was memory delivered through a different channel, and that channel deserves the specification the retrieval stack usually gets. In my fleet the disciplined harness is four concrete things. A versioned instruction layer: the conventions every agent must honor, written as plain, reviewed text that agents load before substantive work, changed through the same review process as code. Scoped context: the handful of documents that govern a task's area, injected at session start, chosen by ownership rather than by similarity search. Deterministic gates: pre-action checks that refuse an edit until the agent has consulted what governs it, because agents do not consult memory spontaneously, mine never did. And independent completion checks that grade the workspace, not the agent's claim about it.
The injected arm's winning sentence is that first element at unit scale: one convention, versioned, explicit, present at the moment of work. It went eight for eight because it never had to win a fight inside the context window; it was the context. The retrieval channel lost that same context-window fight nine consultations at a time. That is the honest content of this note's title. The agent remembers in the harness when the institution's knowledge is engineered into the standing context that surrounds every task, and the store behind it becomes what it should have been from the start: the reviewed source the harness compiles from, not a search index the agent is told to trust mid-flight. The layer this publication chose in earlier notes keeps that role. What changes is where the delivery budget goes first.
Memory use can be optimized by a loop. This fleet has not closed that loop, and says so.
If memory use is the broken half, it can be improved the way the frontier improves it: propose a change to what gets written, condensed, or injected; measure whether graded work outcomes actually improve; keep or retire. Two designs fail before they start. A person reviewing each memory change fails on volume, on the invisibility of single-change effects, and on endurance; the previous note reported that failure operating on this fleet's neighbouring behavioural loop, where acceptance had quietly become one person's attention. And a loop that grades its own changes fails worse; the model-collapse results say how that ends. What worked for the behavioural loop, and is the design here, is selection anchored to outcomes the loop cannot fake, graded verdicts, tests, incidents, real cost, several at once so gaming one does not survive the others, with one standing tripwire: the day the loop's cheap proxy stops tracking the expensive ground truth, it has started gaming itself, and a person amends the rules rather than judging the change. The published instance of gated acceptance is worth one line: a skill library admitting candidates only on net validation gain took an open model from 40.6 to 88.8 percent on a medical-agent benchmark, and its ablation found proposals without the gate no better than no skills at all.
Honesty about status: this fleet has closed exactly one loop of that shape, the behavioural one, three unattended weekly cycles, reported with its ledger in the previous note. The memory loop is designed, not closed; memory effects are finer-grained than habits, so the statistics need more sessions and the honest cadence is slower. Its first gate already exists, though, and it is this note's measurement run continuously: a memory change that cannot beat the harness-only baseline on graded outcomes does not get to stay.
Run this before the next memory dollar. Two weeks, one team, pre-registered.
Everything above compresses into one staffable exercise, sized for one platform team and about two weeks: pick one convention your memory layer is supposed to carry, and make the channels compete for it.
Hold the knowledge constant. Vary only the channel. Grade the work.
One: choose a real task where honoring the convention separates the idiomatic fix from a tempting wrong one, and write a deterministic grader for it, including the masked verdict, works-but-breaks-the-rule. Two: run four arms with the same model, stripped baseline, harness only, the convention as one versioned sentence in standing context, and your live memory layer with consultation forced. Three: write the reading rule before the runs. If the memory arm lands where the harness-only arm lands, your delivery channel is the problem: move the budget to the versioned-context channel, the instruction layer, scoped injection, gates, and completion checks from section v, and let the store be its reviewed source. If the memory arm beats harness-only on graded outcomes, your layer earns its keep; scale the measurement before scaling the spend. Four: either way, keep the exercise running on a cadence, because it is also the acceptance gate any future memory optimization loop needs.
What this note claims, and does not. Measured, on one fleet: the same institutional convention held 8/8 through the versioned-context channel and 1/8 through the live retrieval channel, p = 0.0014; in the controlled follow-up the retrieval layer's marginal value over an equivalent-discipline harness was zero on every metric with retrieval forced and succeeding; and the failure is a use failure, consulted but ignored, not a storage failure. Not claimed: that memory layers are worthless, this was one hard task class where the retrieved convention had to override a strong prior, the aligned-prior case is unmeasured, and cross-route replication, the same design on a second model route, is instrumented but not yet run; nor that the frontier magnitudes transfer off their benchmarks; nor that any memory optimization loop is closed here. The numbers are small and the design is honest about it; that is why the rule above says measure before scaling, in both directions.
The benchmark your memory layer must beat is one honest sentence in the prompt.
Mine lost to that sentence, eight to one, on a task where memory had to override a strong prior, with retrieval working perfectly. If yours has never been asked the question, you are paying for a layer whose value is assumed, not measured. The fix is not better storage and not weight training: it is engineering the channel that won, versioned instructions, scoped context, deterministic gates, independent completion checks, and letting every memory dollar prove itself against that harness.
An independent applied-AI engineering note by Arseny Gorokh. Not an official EPAM publication. The measurement is from the author's own fleet records, June 2026: a pre-registered four-arm design, forty-one real agent sessions, deterministic grading, with the fleet's internal system names withheld by design. The frontier numbers are computed from the cited papers. One task, one model route: treat the magnitudes as corpus-specific and the structure of the argument as the claim.
References
- S. Wu, H. Zhu, Y. Zhang, X. Wang, and S. Yeung-Levy, "AutoMem: Automated Learning of Memory as a Cognitive Skill", arXiv:2607.01224, 2026. The frontier result read in section iv: memory management as a trainable skill; the Table 1 ablation this note's eighty-to-eighty-nine-percent figure is computed from; a 32B open model at the level of frontier systems on three long-horizon game environments.
- "Memory as Action: Autonomous Context Curation for Long-Horizon Agentic Tasks", arXiv:2510.12635, 2025; and "Mem-α: Learning Memory Construction via Reinforcement Learning", arXiv:2509.25911, 2025. The sibling trainable-memory results: a 14B model curating its own working memory matches models sixteen times larger; memory construction trained at thirty-thousand-token scale generalizes past four hundred thousand.
- "Large Language Model Agents Are Not Always Faithful Self-Evolvers", arXiv:2601.22436, 2026. The benchmark-scale version of this note's measured mechanism: agents disregard or misread the condensed experience they wrote for themselves.
- "GRASP: Gated Regression-Aware Skill Proposer for Self-Improving LLM Agents", arXiv:2605.29668, 2026. The published gated-acceptance instance in section vi: proposals without validation were no better than no skills at all; with the gate and a hard regression budget, an open model went from 40.6 to 88.8 percent on MedAgentBench.
- I. Shumailov, Z. Shumaylov, Y. Zhao, N. Papernot, R. Anderson, and Y. Gal, "AI models collapse when trained on recursively generated data", Nature 631, 755-759 (2024). The standing argument against a loop that grades itself.
- C. Packer et al., "MemGPT: Towards LLMs as Operating Systems", arXiv:2310.08560, 2023; and T. Sumers, S. Yao, K. Narasimhan, and T. Griffiths, "Cognitive Architectures for Language Agents", arXiv:2309.02427, 2023. The memory-architecture lineage the trainable-memory work now optimizes over.
- The prior notes in this thread: "Choosing the memory substrate for enterprise agents" (ai-04) and "A memory-and-policy layer above the model" (ai-05), the two notes whose position this measurement audits; "Self-improvement is an actuation problem." (ai-11), the behavioural loop that closed, whose acceptance doctrine section vi extends to memory; and "'It's done.' The most expensive tokens an agent produces." (ai-09), the completion gate whose workspace-state verdicts are one of the non-proxy anchors.