Kai Hirota

8 min read1,802 words

More Reasoning Doesn't Fix Stale Memory

Across every reasoning-budget setting I tested, the benchmark's hardest failure family stayed at zero. What closes it is 40 lines of query-time lookup.

The hardest scenarios in the benchmark are not the ones where the memory has the wrong answer. They are the ones where the memory has the right answer, with the right citation, and still cannot tell that the answer is no longer safe to act on. The benchmark calls this family cascading staleness: the stored answer and its citation are both right, but a premise 1 or more steps back has been superseded. Every system I tested, before adding the fix below, scored 0 on it.

This is the third piece in a 4-part series. The second piece showed that structured access to the store reaches the high 350s out of 395 but misses answers whose premise 3 hops back has changed. This article is about those answers.

Here is the example. The store has an answer claim: Maya Patel has full signing authority at Apple. That claim derives from Maya Patel is chief of staff, which derives from Tim Cook is CEO. Later, the memory learns that Priya Raman became CEO on November 1, 2027. The answer claim has not disappeared, but one upstream premise is no longer current.

Benchmark row E.5

One answer, two different memory states.

Question

What is the signing_authority of Maya Patel?

Expected answer
full
Expected citation
ap_d3
Expected status
POTENTIALLY_STALE

Prompt rule used by the LLM walk

  1. For every doc_id p in c.derived_from, transitively, look up the premise document p.
  2. For each premise, find the current claim for the same entity and predicate at the queried (T_v, T_t).
  3. If that current claim's value differs from p.value, the premise is superseded.
  4. If c is valid and any transitive premise is superseded, set epistemic_status = POTENTIALLY_STALE while still returning c.value.

Evidence chain

Returned claim

ap_d3 / valid now

Maya Patel holds full signing authority for routine corporate matters at Apple.

Direct premise

ap_d2 / depends on ap_01

Maya Patel serves as chief of staff to Apple's CEO Tim Cook.

Transitive premise

ap_01 / superseded

Tim Cook continues as CEO of Apple through most of 2027.

Current successor

ap_03 / current

Apple appoints Priya Raman as Chief Executive Officer effective November 1, 2027.

Row store

No premise walk

fails status
answer
full
primary_document_id
ap_d3
citation_level
3 / 3
epistemic_status
UNVERIFIED

The value and citation are right, but the interface treats the claim as a clean single-source fact.

Sonnet 4.6

Prompted premise walk

passes
answer
full
primary_document_id
ap_d3
citation_level
3 / 3
epistemic_status
POTENTIALLY_STALE

When the prompt surfaces `derived_from` and asks for the transitive walk, the same answer carries the warning.

Row store

Recursive CTE walk

passes
answer
full
primary_document_id
ap_d3
citation_level
3 / 3
epistemic_status
POTENTIALLY_STALE

The storage walk produces the same status without spending tokens or asking a model to reason through the chain.

The same pattern appears outside the executive-delegation chain. Here is the NVIDIA acquisition-freeze row from the corpus, shown with the prior documents the system sees, the prompt contract asking for epistemic status, and the structured answer that passes or fails the audit.

Benchmark rows E.5 / E.6

Asking for status in the prompt is not enough; the read path has to compute it.

Prior ingested documents

Premise

nv_06

The European Commission opens a formal antitrust review of NVIDIA's GPU licensing practices effective August 1, 2027.

predicate: regulatory_status

value: under_antitrust_review

Derived answer

nv_d2

Pursuant to NVIDIA's antitrust review status, an internal acquisition freeze takes effect 2027-08-01 and remains in place pending regulatory resolution.

predicate: acquisition_freeze

value: active

derived_from: nv_06

Successor

nv_07

The European Commission concludes its NVIDIA antitrust review with no enforcement action; status returns to cleared as of February 1, 2028.

predicate: regulatory_status

value: cleared

supersedes: nv_06

Representative prompt contract

You are answering from an auditable memory store.
Use the ingested claim records and their document_id, valid-time interval, transaction time, and derived_from links.
Return answer, primary_document_id, citation_level, and epistemic_status.
epistemic_status must be one of UNVERIFIED, CORROBORATED, CONTESTED, ENRICHED, or POTENTIALLY_STALE.
If the answer claim derives from another claim, transitively inspect each premise at the query valid time and transaction time.
If any transitive premise has a different current value at the query point, keep the answer but set epistemic_status to POTENTIALLY_STALE.

Baseline failure

What is the acquisition_freeze of NVIDIA?

fails status
answer: active
primary_document_id: nv_d2
citation_level: 3
epistemic_status: UNVERIFIED

The answer and citation are right, but the memory state is wrong. The read path returned nv_d2 without walking back to nv_06 and comparing that premise with its current successor nv_07.

Giving the models more reasoning did not fix it.

I ran 3 tests that kept the model and prompt fixed and changed only the reasoning setting. Total scores move, and by a lot: Sonnet 4.6 picks up 83 scenarios when extended thinking is turned on. scores identically across its 3 effort levels; wanders within a 4-scenario band. Cascading staleness stays at 0/29 in every condition.

Within-subject reasoning controls

Scores move with reasoning budget. Cascading staleness doesn't.

GPT-5-mini · reasoning effortE.5 0/29low347medium347high347no change across the sweepSonnet 4.6 · extended thinkingE.5 0/29off276on (8K budget)359+83 with thinking onHoncho 3 Dialectic Agent · reasoning levelE.5 0/29minimal44low45medium44high44max484-scenario spread across all 5 levels395 (full)
3 models, 3 sweeps that vary only the reasoning budget. Total scores move by up to 83 scenarios on the Sonnet 4.6 thinking-on/off pair. Across every setting in every group, cascading staleness stays at 0 out of 29. Without a procedure that walks the chain, more reasoning does not catch it.

However much extra computation the model spends, it does not flag these answers as stale. What closes the family is 40 lines of code that follow links between claims back from the answer at query time. Each link is recorded when a claim is stored. The code is small enough to write in an afternoon, and the score lift is the same on every store that records the links.

Score lift

Before / after the premise walk.

0100200300395Row-store (SQLite)359388+29Mem0359388+29Graphiti354383+29Sonnet 4.6 (thinking on)359388+29before walkafter walk
Headline-corpus scores out of 395, before and after adding a query-time walk over declared derivation links. The lift is consistent across stores because the algorithm is the same; the store only determines how the edges are persisted.

The Walk#

A derived_from link is just an authored field on each claim: when the ingest step records a new claim that was inferred from an earlier one, it writes down which earlier claim it depended on. The procedure is: retrieve the candidate claim. Walk back along those declared derived_from edges. For each premise on the chain, ask the store what the current value is for that (entity, predicate) pair under the query's valid-time and transaction-time bounds. If any premise has been superseded, flip the answer's epistemic status to POTENTIALLY_STALE.

On the SQLite row store, the walk is 2 queries:

-- 1. Every premise behind the answer claim, however many hops back.
WITH RECURSIVE premises(doc_id) AS (
  SELECT premise_doc FROM claim_derivations WHERE derived_doc = :answer_doc
  UNION
  SELECT cd.premise_doc FROM claim_derivations cd
  JOIN premises p ON cd.derived_doc = p.doc_id
)
SELECT c.entity, c.predicate, c.value
FROM premises p JOIN claims c ON c.document_id = p.doc_id;
 
-- 2. For each premise, the value current at the query's 2 time points.
SELECT value FROM claims
WHERE entity = :entity AND predicate = :predicate
  AND t_vs <= :t_v                    -- true in the world at the queried time
  AND (t_ve IS NULL OR t_ve > :t_v)
  AND t_t_ord <= :t_t                 -- known to the memory at the queried time
ORDER BY t_vs DESC, t_t_ord DESC
LIMIT 1;

If any premise's current value differs from the value recorded on the chain, the answer is POTENTIALLY_STALE. This is simplified from the benchmark's reference store, which also handles queries without a time bound and predicates that hold several values at once.

Walking the chain, one premise at a time.

Step 1

Start at the answer claim.

The query returns Maya Patel's signing authority. A safe interface starts a walk here instead of stopping.

Step 2

Hop back along derived_from.

The signing-authority claim was derived from a chief-of-staff claim, which was derived from a CEO claim. The walk follows the declared dependency edges.

Step 3

Compare each premise to the current value.

For every (entity, predicate) on the chain, ask the store: who is current under the query time bounds? Tim Cook was the premise. The current premise is Priya Raman.

Step 4

If any premise was superseded, flip the status.

The answer claim is still returned, but its epistemic status changes from UNVERIFIED to POTENTIALLY_STALE, because its support chain now passes through a superseded premise.

It uses no machine learning, embedding, reranker or calibration: it is a recursive lookup with a deterministic time filter.

4 Implementations, 1 Algorithm#

The procedure transfers cleanly. I implemented it on a SQLite row store as a recursive SQL query, on Mem0 by walking the metadata.derived_from field in Python, on Graphiti by walking the same kind of field on its edge objects, and on the model as a prompt rule with extended thinking turned on (a setting that lets the model spend more tokens on internal reasoning before answering). The same lift appears in all 4.

Same algorithm

4 implementations, 1 walk, the same lift.

The walk

ABCderived_fromderived_from

Substrate

Row-store (SQLite)

recursive SQL query over claim_derivations

before
359
after
388
lift+29

Substrate

Mem0

walk metadata.derived_from

before
359
after
388
lift+29

Substrate

Graphiti

walk EntityEdge.attributes

before
354
after
383
lift+29

Substrate

Sonnet 4.6 (in-context)

prompt rule + extended thinking

before
359
after
388
lift+29
The walk is implemented as a recursive SQL query on SQLite, an in-Python edge walk on Mem0 and Graphiti, and a prompt rule for the Sonnet 4.6 model with extended thinking already on. All 4 close cascading staleness on the headline corpus. The lift is the same because the walk is a read-path algorithm, not a storage-engine feature.

So the fix for cascading staleness is a read-path algorithm, not a storage feature. As long as the write path has authored the dependency edges and the read path is willing to walk them, it does not matter what runs it.

False Alarms#

The naive fix is to flag any claim whose support chain touches a superseded premise. That catches every cascading-staleness case, but it also catches a lot of cases that aren't actually stale.

Concretely: suppose the agent asks "did Maya have signing authority in January?" The support chain still passes through "Tim Cook is CEO", a premise that was later superseded, when Priya Raman became CEO on November 1, 2027. A walk that flags every supersession in the chain would mark the January claim POTENTIALLY_STALE. But for the period being asked about, the chain was sound. The supersession sits in the future relative to the query.

This is the specificity test: 28 scenarios where a real supersession exists, but it lies outside the bitemporal scope the agent is asking about. In the January example, it is outside the valid-time window; in a late-correction case, it may be outside the transaction-time knowledge cutoff. A correct walk leaves these alone.

The denominator differs from the headline stale-case count because the near-misses are a matched real-entity control: 28 real-entity stale cases get 28 near-misses. The headline count is 29 because it also includes the Acme worked-example case.

Valid-time scope

Drag the query bound across the supersession.

UNVERIFIED
Jan 2027Jul 2027Jan 2028Jul 2028

If query bound < supersession (E.6)

The supersession is outside the query's valid-time scope. Status: UNVERIFIED.

If query bound ≥ supersession (E.5)

A premise was superseded inside scope. Status: POTENTIALLY_STALE.

This view moves the query's valid-time bound; the same walk is also scoped by the transaction-time knowledge cutoff. Marking everything stale near a supersession would fail the specificity control: 28 out of 28 near-misses where the supersession exists in the world but lies outside the bitemporal scope the agent was asking about.

So the walk has to be scoped to the query's bitemporal bounds. For each premise, compare against the value that is current for the same (entity, predicate) at the query's valid time TvT_v and known by the query's transaction time TtT_t. A supersession that starts after the period being audited, or that the memory had not learned by the transaction-time cutoff, should not make the January answer stale.

Why Traversal, Not Reasoning#

The sweeps above show that cascading staleness is not a reasoning problem the model could solve with a larger budget. What the family needs is a procedure that walks the chain, and that is a property of the read path rather than of the policy running on top of it.

MRAgent's active-reconstruction result points in the same direction: memory access should be an iterative read procedure whose next step depends on evidence already found, not a passive retrieval call fixed by the original query.[4] The result here is a narrower audit version of that claim. Active traversal can help find the right support, but the benchmark still requires one extra output: whether any premise on the declared derivation path has been superseded under the query's valid-time and transaction-time bounds.

When the LLM Walks the Chain Itself#

If the storage path will not walk the chain, the model has to. I tested this by giving capable long-context LLMs the source documents plus a prompt rule that described the walk, and asked them to execute it themselves. 2 pass/fail tests separate the results: did the system catch the genuinely stale case (sensitivity), and did it leave the alone (specificity)?

Sensitivity vs specificity

Step 1 of 3: All systems

13 systems queued for 2 tests.

Each system in its default setup, plus six variants that walk the dependency chain at query time.

no walk, base
no walk, LLM
walk on storage
walk in prompt (capable)
walk in prompt (over-marks)
2 tests with simple pass/fail outcomes on the 28 matched real-entity pairs. E.5 (sensitivity) asks whether the system flagged the genuinely stale case; E.6 (specificity) asks whether it left the near-miss alone. Storage walks always end up in the top-right cell. Prompt walks depend on how carefully the model can reason: one variant slips into the bottom-right cell because it marks every near-miss stale.

GPT-5-mini and Sonnet 4.6 with extended thinking on match the storage walk on both tests. Sonnet 4.6 with extended thinking off catches the stale cases but also marks every near-miss stale, raising false alarms on cases that aren't actually stale. Smaller models miss both axes. Storage walks always end up in the top-right cell: every stale case caught, every near-miss left alone.

What the Walk Changes#

A concrete failure, using the Apple example above: after the November 1 CEO update, ask “what is the signing_authority of Maya Patel?” The signing-authority claim is in the store with its derived_from link pointing at the chief-of-staff claim, which points back at the Jan 1 “Tim Cook is CEO” premise. The system returns full, with the right citation. The November 1 update where Priya Raman becomes CEO never enters the answer. A reader of the answer cannot tell that the support chain has changed; only the status on the answer should have flipped from UNVERIFIED to POTENTIALLY_STALE.

The lift is about 29 scenarios, and it is the same on every implementation. SQLite row-store 359 → 388, Mem0 A 359 → 388, Graphiti A 354 → 383, Sonnet 4.6 (with extended thinking) 359 → 388. Same procedure, same delta, on 3 storage backends and an in-context LLM. The lift depends on the corpus, not on the backend.

The Sonnet path has 2 effects, not 1. Sonnet 4.6 in-context starts at 276/395. Turning on extended thinking lifts that to 359/395 (Δ83) without touching the stale cases at all; the model gets better at the rest of the corpus. Adding the premise-walk prompt on top contributes the same Δ29 the storage walks deliver elsewhere, taking it to 388. The walk and the reasoning budget do different things, and Sonnet needs both to reach 388.

The walk adds status, not accuracy. For systems whose base answer+citation score was already high (Mem0 A: 389/395 on answer+citation), the +walk answer+citation barely moves (389 → 389), while the full-rubric jumps (359 → 388). The walk is closing the gap between “I have the right answer” and “I can also say whether the answer is still safe to use”. It fixes epistemic status, not retrieval.

The next article puts every system on 1 chart and sends the same questions to the managed products you can buy, not only to libraries run locally.

What the Walk Depends On#

The walk works conditional on a write path that authors the derivation edges. The benchmark assumes that input. If your ingest step does not preserve derived_from, the read-path procedure has nothing to walk.

Scope

What the benchmark measures, and what it doesn't.

Write pathout of scopedocument arrivesextraction (LLM or other)claim + derived_from emittedpersisted to storestoreRead pathmeasuredquery receivedretrieve candidate claimwalk derived_from closurecompare each premise to currentemit answer + status
The benchmark assumes the write path has already authored the derivation edges. It measures whether the read path uses them. Extraction quality is a separate problem; better extraction does not help if retrieval never walks the chain.

Write-path extraction quality is a separate problem, with its own literature. The benchmark shows that, given the edges, the read path is a small piece of code that works on any store. It is not yet standard in agent-memory APIs.

The failure is invisible from the answer alone: the answer and the citation are right, only the status is wrong, and most memory APIs have no place to put a status. The benchmark measures that status directly. The framing piece draws the parallel to the metacognition argument for parametric models: in both settings, reliability hinges on a per-answer faithful signal, not on aggregate accuracy.

Limitations#

  • Depends on write-path quality: missing provenance links reduce recall of stale-chain detection.
  • Needs stable claim IDs across ingestion, compaction, and deduplication.
  • Deep or noisy provenance graphs need cycle handling, fan-out caps, and observability around skipped links.
  • Benchmark scores are controlled evaluations; production distributions and noise can differ.

References#

  1. Rasmussen, P., Paliychuk, P., Beauvais, T., Ryan, J., & Chalef, D. (2025). Zep: A Temporal Knowledge Graph Architecture for Agent Memory. arXiv:2501.13956.
  2. Chhikara, P., Khant, D., Aryan, S., Singh, T., & Yadav, D. (2025). Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory. arXiv:2504.19413.
  3. Marković, V., Obradović, L., Hajdu, L., & Pavlović, J. (2025). Optimizing the Interface Between Knowledge Graphs and LLMs for Complex Reasoning. arXiv:2505.24478.
  4. Ji, S., Li, Y., & Hooi, B. (2026). Memory is Reconstructed, Not Retrieved: Graph Memory for LLM Agents. arXiv:2606.06036.