Nine Shipping Products, One Blind Spot

Nine memory products you can deploy today were asked questions whose answers rest on a premise that had been superseded. All nine scored zero on that family.

Not because they failed to find the claim. Several found it perfectly well. They returned it without any signal that the evidence underneath it had moved — because most memory APIs have no field in which to put one.

This is a companion to the premise-walk article, which shows that closing this family takes about forty lines of query-time traversal. The question here is narrower and more practical: the products are shipping, some of them market exactly this capability, so what do their own native interfaces actually resolve?

The Full Scoreboard#

Pulling all the systems onto one page: where does each one end up, and what does the read-path walk lift do to the systems that admit it?

The DuckDB row at the top is the interface specification — a roughly fifty-line SQL store implementing the formal validity predicate. It defines what 395/395 means; it is not a competitor. Everything below it is a system you might actually deploy. The dashed extensions show the read-path walk lift on the systems where I measured it.

The strongest structured and no-LLM non-walk rows in the chart cap near 359/395 — a 36-scenario gap dominated by the cascading-staleness family (cascading staleness: 29 scenarios). The lower-scoring NL and agentic-memory rows fail additional retrieval or citation axes before the walk even matters.

The score-vs-cost Pareto frontier is the no-LLM walk. The highest-scoring non-reference row on the board (388/395) is also the cheapest: row-store SQLite + recursive-CTE walk runs in under a millisecond per query at zero tokens. Every other row in the 345–388 band pays more and scores less. Mem0 and Graphiti on their structured paths reach 359 and 354 on embedding-only budgets at around 100 ms per query; the LLM-dispatched variants reach 357 and 345 at 110 ms and 2.4 s; GPT-5.5 in-context reaches 347 at roughly 5 s. That is more than three orders of magnitude in latency across rows separated by about forty scenarios. The leverage of the walk is that it doesn't depend on the LLM at all.

Do the Hosted Products Do Better?#

The rows above run the open-source libraries locally, behind a thin wrapper that applies the benchmark's time-and-evidence filter. The fair next question is whether the managed products — the ones you would actually buy — do better through their own native interfaces. So I added three hosted rows: Zep Cloud under edge-scope date filters and under its scope=auto context block, and Mem0 Platform v3 on its temporal-reasoning surface.

They do not. The native surfaces land far below the same libraries run locally behind the benchmark's own filter — Zep Cloud 160/395 and Mem0 Platform v3 22/395, against 354 and 359 — and the reason is the more interesting part.

Zep markets the surface as “context you can trace, filter, and trust”, and the first two of those are real. Every stored fact links back to the source episode it was read from, and you can filter retrieval by valid_at / created_at date ranges and by metadata such as a verified flag. That is exactly why Zep Cloud retrieves the correct claim on ~309/395 — better than any other deployed comparator in the chart. But trust under change is a separate primitive from trace and filter. Strong retrieval does not confer bitemporal-audit completeness: only 160 of those 309 survive the full rubric, and the dashed remainder is the citation and epistemic-status axes. The scope=auto context block retrieves marginally more (318) but audits less (121), because its cross-scope summary underserves structured citation.

And on the hardest family, the hosted products fail exactly the way everything else does. The shared blind spot spans nine product surfaces — Graphiti and Mem0 open-source, the hosted Zep Cloud and Mem0 Platform v3 current products, plus TencentDB-Agent-Memory, Cognee, Honcho 3, Supermemory, and Hindsight. The hosted Zep Cloud surface is the sharpest illustration: its native date filters retrieve the stale-premise answer on most instances, and it still returns UNVERIFIED, never POTENTIALLY_STALE. The products are not failing to find the claim. They are returning it without a signal that its support has moved.

The Temporal-Reasoning Surface#

Two Mem0 posts bear on this directly: Introducing the Token-Efficient Memory Algorithm (April 16, 2026) and The Token-Efficient Memory Algorithm Now Has Temporal Reasoning (May 14, 2026). The May post adds a state-key + event_end model for ongoing facts — a partial valid-time analogue, with write-time temporal metadata and reranking (rather than filtering) at read time.

The work overlaps with the temporal half of this benchmark's contract but does not subsume it. Mem0's update tracks one clock; the benchmark requires two (valid time and transaction time, so retroactive corrections remain distinguishable from real-time updates). It also has no documented slot for per-claim epistemic status or for the derivation-closure walk that closes the cascading-staleness family. The May post adds a memory decay signal alongside the state-key model — recent memories are boosted up to 1.5× and stale ones dampened to 0.3× at ranking time — but that is a recency knob on retrieval, not a propagation of supersession through a premise chain; a dampened claim is still returned, just lower, and still without a status. The May post's own numbers are consistent with that boundary: +3.8 points on LongMemEval temporal reasoning, but −2.6 points on the knowledge-update category — reranking surfaces a newer dated instance, but does not, by itself, propagate a status change through a premise chain.

Mem0 documents both the temporal-reasoning surface and memory decay as platform-only features on top of the open-source base algorithm, so they are exactly what the Mem0 Platform v3 (hosted) row above evaluates — 22/395, with cascading staleness still 0/29. The open-source Mem0 rows reflect the base-algorithm version; the hosted row is the temporal-reasoning platform. Both land on the same side of the boundary.

Limitations#

• Each hosted row is a single configuration of a managed product, queried through the native interface its documentation describes; a different configuration may score differently.
• Managed products ship continuously, so these are point-in-time measurements of surfaces that move.
• Benchmark scores are controlled evaluations; production distributions and noise can differ.

References#

1. Rasmussen, P., Paliychuk, P., Beauvais, T., Ryan, J., & Chalef, D. (2025). Zep: A Temporal Knowledge Graph Architecture for Agent Memory. arXiv:2501.13956.
2. Chhikara, P., Khant, D., Aryan, S., Singh, T., & Yadav, D. (2025). Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory. arXiv:2504.19413.
3. Marković, V., Obradović, L., Hajdu, L., & Pavlović, J. (2025). Optimizing the Interface Between Knowledge Graphs and LLMs for Complex Reasoning. arXiv:2505.24478.

Systems referenced#

Each row is run against its public production interface; configuration detail is in the paper.

• Graphiti (OSS) — temporal knowledge-graph memory library for LLM agents, run locally with a wrapper-side filter. github.com/getzep/graphiti.
• Zep Cloud (hosted) — the managed product from the same team, queried through its native graph.search() valid_at / created_at date filters and its scope=auto context block. getzep.com.
• Mem0 (OSS) — write-time-extraction memory layer (local), with a structured API and an NL retrieval call. mem0.ai · github.com/mem0ai/mem0.
• Mem0 Platform v3 (hosted) — the managed product's temporal-reasoning surface (state-key + event_end, memory decay), documented as platform-only. mem0.ai.
• TencentDB-Agent-Memory — local SQLite-backed layered memory with L0 conversation search, L1 structured memories, and traceable persona / scenario summaries. github.com/Tencent/TencentDB-Agent-Memory.
• Supermemory — write-time extraction into a memory graph with documented update / extend / derive relation labels. supermemory.ai.
• Hindsight — typed four-network agent memory with retain / recall / reflect operations. github.com/vectorize-io/hindsight.
• Cognee — graph + vector memory engine with GRAPH_COMPLETION, CHUNKS, and TEMPORAL search modes. github.com/topoteretes/cognee.
• Honcho 3 — managed personal-memory service driven through a Dialectic Agent. honcho.dev.