7 min read1,473 words
9 Shipping Products, 1 Blind Spot
Every deployed memory system I tested returns the claim with no signal that its evidence has changed, including the one that markets provenance.
9 memory products you can deploy today were asked questions whose answers rest on a premise that had been superseded. All 9 scored zero on that family.
Several found the claim. They returned it without any signal that the evidence behind it had changed, because most memory APIs have no field in which to put one.
This is the last piece in a 4-part series on a benchmark I built to test whether agent memory
notices when the evidence behind a stored answer changes. The
first piece introduces it. Each of its 395
scenarios is graded on the answer, the citation chain, the epistemic status, and whether ingest
finishes in bounded time; a scenario passes only when all 4 pass. The status that matters here is
POTENTIALLY_STALE: the answer may be right, but a premise behind it has been superseded.
The third piece shows that closing this family takes about 40 lines of code that follow links between claims back from the answer at query time. This one asks what the shipping products resolve through their own native interfaces, given that some of them market this capability.
The 0 is partly a property of the interface: a product with no status field cannot return
POTENTIALLY_STALE, however well it retrieves. The fix also relies on links between claims that
the benchmark's data declares, and a product would have to record those links itself before
it could follow them. Neither point makes the 0 less of a problem. An agent using these products
has no way to tell a current answer from one whose premise has changed.
The Full Scoreboard#
Every system on 1 chart, with the lift from the walk on the systems that can run it.
Headline scoreboard
All evaluated systems, sorted by full-rubric pass rate.
The DuckDB row at the top is a roughly 50-line SQL reference store that shows what a perfect 395/395 looks like. It is not a competitor. Everything below it is a system you might actually deploy. The dashed extensions show the walk's lift on the systems where I measured it.
Without the walk, the best rows in the chart cap near 359/395, a 36-scenario gap dominated by the cascading-staleness family (cascading staleness: 29 scenarios). The lower rows also miss answers or citations, so they fail before the walk would matter.
The best score is also the cheapest. The highest-scoring non-reference row on the board (388/395) is SQLite with the walk written in SQL. It runs in under a millisecond per query at zero tokens. Every other row in the 345–388 band pays more and scores less. Mem0 and Graphiti on their structured paths reach 359 and 354 on embedding-only budgets at around 100 ms per query; the LLM-dispatched variants reach 357 and 345 at 110 ms and 2.4 s; GPT-5.5 in-context reaches 347 at roughly 5 s. That is more than 3 orders of magnitude in latency across rows separated by about 40 scenarios. The walk does not depend on the LLM at all.
Do the Hosted Products Do Better?#
The rows above run the open-source libraries locally, behind a thin wrapper that
applies the benchmark's time-and-evidence filter. The next question is
whether the managed products, the ones you would actually buy, do
better through their own native interfaces. So I added 3 hosted rows: Zep Cloud
under edge-scope date filters and under its scope=auto context block, and Mem0 Platform v3
on its temporal-reasoning surface.
Their native surfaces score far lower than the same libraries run locally behind the benchmark's own filter: Zep Cloud 160/395 and Mem0 Platform v3 22/395, against 354 and 359.
Hosted current products
Strong retrieval, weak audit, on the native interface.
valid_at / created_at date filters and provenance-to-source retrieve the correct claim on ~309/395, but only 160 survive the full audit; the dashed remainder is the citation and epistemic-status axes the managed surface does not expose. Run locally with a wrapper-side filter (Mem0 OSS Mode A), retrieval and audit nearly coincide.Zep markets the surface as “context you can trace, filter, and
trust”,
and the first 2 of those hold up. Every stored fact links back to the source
episode it was read from, and you can filter retrieval by valid_at /
created_at date ranges and by metadata such as a verified flag. That is
why Zep Cloud retrieves the correct claim on ~309/395, better
than any other deployed comparator in the chart. But trust under change is a
separate primitive from trace and filter. Strong retrieval does not confer
bitemporal-audit completeness: only 160 of those 309 survive the full rubric,
and the dashed remainder is the citation and epistemic-status axes. The
scope=auto context block retrieves marginally more (318) but audits less
(121), because its cross-scope summary gives weaker structured citations.
On the hardest family, the hosted products fail the same way as everything
else. The shared blind spot spans 9 product surfaces:
Graphiti and Mem0 open-source, the hosted Zep Cloud and Mem0 Platform v3 current
products, plus TencentDB-Agent-Memory, Cognee, Honcho 3, Supermemory, and
Hindsight. The hosted Zep Cloud surface is the sharpest illustration: its native
date filters retrieve the stale-premise answer on most instances, and it still returns
UNVERIFIED, never POTENTIALLY_STALE.
The Temporal-Reasoning Surface#
2 Mem0 posts are relevant here: Introducing the Token-Efficient Memory
Algorithm
(April 16, 2026) and The Token-Efficient Memory Algorithm Now Has Temporal
Reasoning
(May 14, 2026). The May post adds a state-key + event_end model for
ongoing facts: a partial valid-time analogue, with write-time temporal
metadata and reranking (rather than filtering) at read time.
The work covers part of the temporal half of this benchmark's contract, not all of it. Mem0's update tracks one clock; the benchmark requires 2 (valid time and transaction time, so retroactive corrections remain distinguishable from real-time updates). It also has no documented slot for per-claim epistemic status or for the derivation-closure walk that closes the cascading-staleness family. The May post adds a memory decay signal alongside the state-key model (recent memories are boosted up to 1.5× and stale ones dampened to 0.3× at ranking time), but that is a recency knob on retrieval, not a propagation of supersession through a premise chain; a dampened claim is still returned, just lower, and still without a status. The May post's own numbers are consistent with that boundary: +3.8 points on LongMemEval temporal reasoning, but −2.6 points on the knowledge-update category: reranking surfaces a newer dated instance, but does not, by itself, propagate a status change through a premise chain.
Mem0 documents both the temporal-reasoning surface and memory decay as platform-only features on top of the open-source base algorithm, so they are what the Mem0 Platform v3 (hosted) row above evaluates: 22/395, with cascading staleness still 0/29. The open-source Mem0 rows reflect the base-algorithm version; the hosted row is the temporal-reasoning platform. Neither closes the cascading-staleness family.
What It Takes to Close the Gap#
The premise-walk article runs the same fix on Mem0 and Graphiti. It needs 3 things:
- Record the links at ingest. When a claim is derived from earlier claims, store their IDs
with it (
metadata.derived_fromon Mem0, edge attributes on Graphiti). The IDs have to survive compaction and deduplication. - Walk the links after retrieval. For each premise, look up the value current at the query's valid time and transaction time, and compare it with the value on the chain.
- Return a status with the answer. If any premise changed, the answer goes back as
POTENTIALLY_STALE, so the agent can see it.
Limitations#
- Each hosted row is a single configuration of a managed product, queried through the native interface its documentation describes; a different configuration may score differently.
- Managed products ship continuously, so these are point-in-time measurements of surfaces that move.
- Benchmark scores are controlled evaluations; production distributions and noise can differ.
References#
- Rasmussen, P., Paliychuk, P., Beauvais, T., Ryan, J., & Chalef, D. (2025). Zep: A Temporal Knowledge Graph Architecture for Agent Memory. arXiv:2501.13956.
- Chhikara, P., Khant, D., Aryan, S., Singh, T., & Yadav, D. (2025). Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory. arXiv:2504.19413.
- Marković, V., Obradović, L., Hajdu, L., & Pavlović, J. (2025). Optimizing the Interface Between Knowledge Graphs and LLMs for Complex Reasoning. arXiv:2505.24478.
Systems referenced#
Each row is run against its public production interface; configuration detail is in the paper.
- Graphiti (OSS): temporal knowledge-graph memory library for LLM agents, run locally with a wrapper-side filter. github.com/getzep/graphiti.
- Zep Cloud (hosted): the managed product from the same team, queried through its native
graph.search()valid_at/created_atdate filters and itsscope=autocontext block. getzep.com. - Mem0 (OSS): write-time-extraction memory layer (local), with a structured API and an NL retrieval call. mem0.ai · github.com/mem0ai/mem0.
- Mem0 Platform v3 (hosted): the managed product's temporal-reasoning surface (state-key +
event_end, memory decay), documented as platform-only. mem0.ai. - TencentDB-Agent-Memory: local SQLite-backed layered memory with L0 conversation search, L1 structured memories, and traceable persona / scenario summaries. github.com/Tencent/TencentDB-Agent-Memory.
- Supermemory: write-time extraction into a memory graph with documented
update/extend/deriverelation labels. supermemory.ai. - Hindsight: typed 4-network agent memory with
retain/recall/reflectoperations. github.com/vectorize-io/hindsight. - Cognee: graph + vector memory engine with
GRAPH_COMPLETION,CHUNKS, andTEMPORALsearch modes. github.com/topoteretes/cognee. - Honcho 3: managed personal-memory service driven through a Dialectic Agent. honcho.dev.