The benchmark that ranks substrates can hide a confident wrong answer.
If you are choosing a memory substrate for agents, the failure that costs you is not a missing fact, it is a confident wrong one, and the benchmark you would normally trust to rank substrates can hide exactly that.
This note is structured as a deployment-decision memo, not a benchmark recap. Two purpose-built substrates run against a filesystem baseline on one SDLC-shaped corpus, scored by two independent judges in opposite orderings to rule out position bias. The four findings below are the headline; the body is the supporting argument. The frontier layer keeps publishing capability jumps, Google Research's TurboQuant (ICLR 2026, 3-bit KV-cache quantisation, about 8x H100 throughput) is a recent one with no managed-inference SLA from a major vendor as of this writing, and consumer products convert a subset into features (Claude memory, ChatGPT's cross-session dossier). Neither layer ships the tenancy, audit, schema-stability, or cost-bound contract an enterprise substrate has to honour. This report closes that gap on one capability: long-term memory for coding-orchestration agents.
Production-weighted retrieval separates the two substrates by 36 points; uniform weighting is a tie.
The chunk-text retriever (LightRAG) outperformed the entity graph (Graphiti) on this corpus's observed query distribution once the eight test families were weighted to production frequency. Under uniform weighting the two were statistically tied, with a confidence interval that crossed zero. The methodology finding, the one that transfers, is that the weighting choice decides the outcome; the 36-point magnitude is a function of this corpus and its production profile, and another team's corpus will produce a different gap. Re-derive the weighting on your own corpus before treating the magnitude as portable.
Two corpus shapes, one retrieval problem.
Across enterprise-agent engagements the memory-substrate question recurs because agents are useful only when they retrieve the right knowledge at the right moment. The corpus shape varies; the underlying problem does not. Two classes cover most of the work. Class A is the SDLC corpus: ADRs, design records, ticket history, post-incident reviews, runbooks, architecture invariants, queried by its present. Coding-orchestration agents have to remember what was decided three turns ago, hold a project-wide invariant across files, and resist drifting into a different solution because the conversation accumulated noise. The hallucination cost is a reversed architectural decision. Class B is the business or domain corpus: case files, regulatory references, product knowledge, customer history, where high volume meets high query specificity and the cost of a wrong answer is that it gets acted on at a client's expense.
Both classes share the same retrieval problem: fetch the relevant knowledge with bounded hallucination, when the agent needs it. The substrate-selection question is identical; only the corpus shape and the production query distribution differ. The empirical work here is on Class A; the substrate-pairing pattern transfers to Class B with re-measured magnitudes. Three structural choices, in combination, define a memory architecture: what is retained (turns, file chunks, summarised sessions, decision artefacts), how retention is keyed (recency window, semantic similarity, tags, graph traversal, deterministic indexing), and what gets surfaced per turn (the slice the executor actually sees on a given call). LightRAG and Graphiti differ on all three. Enterprise teams do not buy a model; they buy a system whose memory architecture decides whether a multi-day workflow holds together.
Five criteria, weighted, stated up front.
A substrate that survives the first production query is one selected against named criteria, not against taste. Stating the criteria first is what makes a rejection reproducible by another reviewer applying the same bar to the same project. The five below are calibrated to one delivery context, a Python-first stack, an EPAM DIAL gateway, an MCP-compatible agent surface; a team with a TypeScript-only stack would weight integration surface differently, and a team with a different gateway would re-state operational fit. The discipline is to publish the bar; the contents are domain-specific.
| criterion | what it asks | weight | pass bar |
|---|---|---|---|
| Integration surface | Python-callable read path, MCP-compatible agent interface, multi-tenant per-workspace mode. | 0.25 | all three present. |
| Install hygiene | no post-install scripts that patch sibling packages, modify the host, or silently fall through on error. | 0.20 | clean install. |
| Test coverage | a non-trivial suite that exercises the API the consuming stack would call; wired-but-empty fails. | 0.20 | suite present, non-trivial. |
| Maintenance posture | more than one active contributor, a sustainable cadence, age enough to surface common bugs. | 0.15 | 2-plus contributors, 6-plus months. |
| Operational fit | model calls through the sanctioned gateway, standard telemetry and secrets, no bespoke infra. | 0.20 | gateway-routed; standard secrets. |
Write the bar before you evaluate candidates. A bar applied post-hoc rationalises a decision already made; a bar applied a priori constrains the decision and is reproducible by the next reviewer who runs the same scoring against the same project. The methodology records how each weight was derived.
Three substrates, eight task families, two judges in opposite orderings.
The empirical design served one question: which substrate performs best on the SDLC corpus's actual production query distribution, evaluated so as to rule out the most common position-bias failure mode in LLM-judge benchmarks. Three substrates were measured. The chunk-text retriever, tested as LightRAG (HKU, arXiv:2410.05779), splits documents into chunks and returns the literal source text with its path attached: mature, low-overhead, predictable. The entity graph, tested as Graphiti (Zep, arXiv:2501.13956), runs an LLM extraction step that pulls entities and relations on ingest, stamps nodes with valid-from and superseded-on, and returns node summaries plus relations. The filesystem baseline is the trust-of-last-resort: keyword extraction, recursive grep, the first matched files' head to the judge, the fallback an engineer takes when no purpose-built substrate is available. A fourth candidate, Mem0, was screened out as the same architectural class as Graphiti, an extraction-mediated semantic store with an LLM-rewrite step at ingest, so a second sample of the same approach was not informative.
| task family | what the agent is trying to do | prod freq |
|---|---|---|
| Current canonical text | read the literal current rule, invariant, or decision for X. | ~50% |
| Handoff continuity | where did the previous session leave off, and what is next. | ~20% |
| Entity disambiguation | which of several same-named entities is meant here. | ~10% |
| Temporal supersession | what changed between two points in time, what deprecated what. | ~4% |
| Multi-hop traversal | which investigations led to this decision, what chain supports this rule. | ~4% |
| Knowledge updates | what the new doctrine imposed that the old one did not. | ~4% |
| Abstention on false premise | the question cites a rule that does not exist; refuse to answer. | ~4% |
| Link-graph traversal | which other documents link to this one. | ~4% |
Sixty-four paired questions ran through every pair of substrates. Each retrieval was scored by two independent judges, chosen from different model families so family-specific preferences would surface as judge disagreement rather than silent bias, and the same question was shown to each judge twice with the two candidate retrievals in opposite orderings. A pair only wins when the judge prefers the same candidate in both orderings; position-swap disagreement collapses to a tie. The rule-change threshold was pre-registered before any retrieval ran: a 20-percentage-point absolute gap, a 2x difference in fabrication rate, or a confidence interval that excludes zero. When you run your own canary, do not skip the position-swap: over-counting ties is the feature that forces the headline to survive judge order.
Uniform said tie. Production weighting said 36 points.
The retrieval-quality comparison between the two purpose-built substrates went in opposite directions depending on how the eight task families were combined into one aggregate. Which one is right is a question about the production workload, not about statistics. A uniform-weighted aggregate counts every family equally; a frequency-weighted aggregate counts each by how often it shows up in real production queries. The two often disagree on which substrate wins, and the disagreement is usually the finding. Under uniform weighting, LightRAG and Graphiti were tied: a small win-rate gap with a confidence interval that crossed zero, the number a report stopping at the uniform aggregate would publish. Re-weighted to the production distribution from the prior section, LightRAG beat Graphiti by a wide margin. The two families that carry 70 percent of production weight on this corpus, current canonical text and handoff continuity, are the two where chunk-text wins decisively; the one family where the entity graph wins, entity disambiguation, carries 10 percent.
| substrate | win (weighted) | fabrications | bar | standing |
|---|---|---|---|---|
| LightRAG chunk-text | 68% | 0 / 128 | 5 / 5 | production-weighted win; clears the bar; the default. |
| Graphiti entity-graph | 32% | 4 / 128 | 5 / 5 | tied under uniform, below under production; entity-disambiguation side path. |
| Filesystem baseline | below both | 0 / 128 | n/a | intentionally naive; the trust-of-last-resort comparison. |
The chunk-text substrate returns the literal source sentence, which is what current-canonical-text queries actually want; the entity graph runs an LLM extraction step at ingest that rewrites node summaries, so the literal phrasing is lost by the time the agent retrieves it. Substrate architecture sets the ceiling for each family; the family weighting decides which ceiling matters for the aggregate. One caveat on transfer: a corpus where entity disambiguation dominates, a customer-support knowledge base where the same product is named differently across regions, would re-weight in favour of the entity graph. The pattern transfers, weight by your own production distribution and not by uniform; the magnitude does not transfer without re-measurement. The pre-measurement reading of the published vendor numbers had the entity graph as the canonical substrate on its bi-temporal differentiation; the frequency-weighted canary inverted that in the first pass, and the retraction is itself a finding: an adoption bar that catches errors before they ship is worth more than a decision that was right on the first try.
Zero versus four out of 128, small in absolute, structural in shape.
Counting fabricated answers is not the most powerful retrieval-quality discriminator at this sample size; the shape of the failure mode behind the count is the part that should change a deployment decision. A retrieval is hallucinated when it returns content that does not exist in any source document, and incorrect when it returns the wrong source document. The distinction matters because an incorrect-but-real source can be cross-checked, and a fabricated one cannot. Across 128 paired appearances per substrate, LightRAG produced zero hallucinated retrievals; Graphiti produced four, about 3 percent; the filesystem baseline produced zero, because its failure mode is the wrong file head, incorrect-but-real, not a fabrication category.
All four invented relations land on the family the bi-temporal metadata was meant to help.
Four out of 128 is too small to clear the rule-change threshold as a standalone statistical claim, but it names the structural difference cleanly. All four of Graphiti's hallucinated outputs were on temporal-supersession queries. The extraction step occasionally produces a "fact F1 was superseded by fact F2 at time t" relation when no source document records the supersession. The metadata that was supposed to be the differentiator is the surface area where the failure mode lives. A chunk-text pipeline has no extraction step on the read path, so the source text either exists or it does not.
This does not say Graphiti is unsafe at scale; four out of 128 cannot be generalised that strongly. It says that under autonomous-writeback paths, where a single fabricated relation gets acted on, the substrate whose failure mode is the wrong chunk has a safer profile than the substrate whose failure mode is an invented relation. The first is recoverable on read; the second is not. For review-gated assist paths with explicit approval before any write, the gap is much smaller. So restrict autonomous-writeback paths, where the agent drafts a decision document, files a ticket, or generates a handoff, to the substrate whose failure mode is the wrong source chunk. These two failure shapes, wrong chunk for the chunk-text retriever and invented relation for the extraction-mediated graph, are the part of the comparison that outlives the specific numbers.
Same extra context, opposite signs across executor tiers.
The structural finding from the closed-loop pilot, the part that does not appear in any vendor's published numbers, is that the value of extra structural context, the bi-temporal valid-from and superseded-on metadata, flips sign across two executor capability tiers tested on the same workload. The counts in this section come from a pilot run, eight tasks per cell, three conditions. Two cells per executor is not enough to claim a sign-flip as a stable property; treat the direction as motivation for the structural claim, not as a per-tier default recommendation. The full closed-loop suite at a defensible cell count is the next iteration of this work.
| executor | tier | effect of bi-temporal context |
|---|---|---|
| Gemini 2.5 Flash | smaller / cheaper | stale-fact recall improves. |
| Claude Sonnet 4.5 | more-capable | stale-fact recall degrades. |
The cheaper executor benefits from explicit supersession metadata because without it the model picks a plausible older fact when no recency signal is in scope. The more-capable executor sometimes re-quotes the older fact because the metadata told it the older fact existed at all; the explicit valid-from and superseded-on payload broadens the candidate set the stronger executor considers, and it occasionally picks the wrong one. Two checkpoints from two model families is not sufficient to fully separate capability-tier effects from family-specific effects; the conservative reading is that capability tier matters, family may also matter, and the interaction is worth measuring per pair. It is sufficient to motivate the structural claim: the value of extra context is conditional on the executor consuming it, not an absolute property of the substrate. For platform teams setting defaults, do not enable bi-temporal context as a global default; leave the payload configurable rather than hard-coded, so the cell-counted re-measurement can drive the default later.
A fourth candidate fails the bar; chunk-text becomes the default.
Two decisions land here. The first is a candidate rejected against the named bar before any retrieval canary cost was spent. The fourth candidate considered, Tencent DB Agent-Memory, the most architecturally distinct option in the set, a four-layer local-memory pipeline with a task-state working memory, was publicly unevaluated outside its own README at evaluation time, so the bar check below is the first published external read on record. Against the five criteria: integration surface failed, distributed primarily as a TypeScript and Node package against a Python-first stack; install hygiene failed, the post-install step rewrites a sibling package's runtime files and silently falls through on error; test coverage failed, the harness is wired but a search for test files in the published source returns zero; operational fit failed, no MCP server surface, no graph database, no fit with the per-workspace HTTP server pattern or with gateway-routed model calls. Maintenance posture was partial, about six weeks old at most recent release with two contributors. Four of five fails, one partial. The decision is reproducible against the stated bar. "Did not fit this adoption bar" is not the same as "is a bad project"; the rejection is re-evaluable when the candidate ships material changes against the failing criteria. The claims ledger records the evidence.
The entity graph leaves the default deployment path; the bar is the durable artefact.
Graphiti cleared integration, install, test, and maintenance criteria under the bar. The production-weighted retrieval gap plus the autonomous-writeback failure-mode shape place it below the operational-fit bar for default deployments. It is retained for the narrow entity-disambiguation side path where its architecture wins; the default path no longer routes to it. Retirement is what the comparison produced, not an operational call made separately.
So the four DIAL-routed defaults for the next deployment: chunk-text retrieval is the default substrate for SDLC-shaped corpora, with the entity graph a side path for entity-disambiguation queries rather than a switch of the default; treat bi-temporal context as a configurable per-workspace toggle, not a global default, until a cell-counted re-measurement confirms the per-tier shape; route autonomous-writeback through the substrate with no extraction step on read, while review-gated assist paths can use either; and route model calls through your sanctioned-inference gateway in production, verifying the rankings on the path you will deploy. Encode these as defaults in the per-workspace configuration so the right choice is the easy choice and the wrong one an explicit opt-out. The read-path MCP bridge used here, workspace-aware routing to LightRAG with a host-scoped allowlist, is open-source at github.com/agorokh/agentic-memory-mcp (Apache-2.0), and the canary harness is published so the next team can re-instantiate it on their corpus in a day.
An independent applied-AI engineering note by Arseny Gorokh. Not an official EPAM publication. EPAM's DIAL and the named substrates are the public subject; the corpus, the canary harness, and the adoption bar are the author's own. The limits bound the claim: one corpus class and one work instance, about 130 SDLC documents; eight paired questions per family, enough to detect large effects only; a deliberately naive filesystem baseline; two-judge position-swap consensus that inflates ties; one provider path during evaluation; a tier-interaction finding resting on two executor checkpoints at pilot N; an adoption-bar verdict that is project-state-as-of-evaluation-time; and no cost-per-decision number. The 36-point gap, the 0-versus-4 fabrication count, and the stale-fact inversion magnitude do not transfer without re-measurement. Re-run the canary on your own corpus before treating any magnitude as portable.
References
- LightRAG (HKUDS), the chunk-text retriever tested here, github.com/HKUDS/LightRAG and arXiv:2410.05779: documents split into chunks, embedding plus keyword index, literal source text returned.
- Graphiti (getzep), the entity graph with bi-temporal metadata, github.com/getzep/graphiti and arXiv:2501.13956: LLM extraction at ingest, nodes stamped valid-from and superseded-on.
- Tencent DB Agent-Memory, the fourth candidate rejected at the bar check before any canary ran; named in the body so the rejection is auditable against the same five criteria.
- Mem0 (arXiv:2504.19413) and the Zep rebuttal on LoCoMo, which dispute each other by 10 to 30 percentage points depending on whose methodology is used; screened out as the same architectural class as the entity graph.
- The production-weighting method: a frequency-weighted aggregate over test families derived from the observed production query distribution, the load-bearing choice that separates the substrates where uniform weighting calls a tie.
- The unbiased-evaluation GraphRAG paper (arXiv:2506.06331), whose position, length, and trial-bias controls motivated parts of the position-swap judging design.
- Google Research, TurboQuant (ICLR 2026), 3-bit KV-cache quantisation at about 8x H100 throughput: the frontier-versus-enterprise framing, a capability jump with no managed-inference SLA from a major vendor as of this writing.