Kai Hirota

7 min read1,590 words

When Remembered Facts Outlive Their Evidence

Why reliable agent memory needs to reconstruct not just answers, but the evidence and status those answers carry under change.

An agent with memory stores facts from earlier sessions and answers from them later. A stored fact can still be retrieved, with its source, after the evidence behind it has changed. I built a benchmark of 395 scenarios to test whether memory systems notice that change. This is the first of 4 articles on what it found.

One memory product that markets itself on provenance was asked for a value whose supporting premise had been superseded 4 sessions earlier. It found the right claim (better retrieval than any other deployed system I tested), returned it with a working link back to the source document, and labelled it UNVERIFIED.

The label was accurate but told the user nothing useful. Nothing in the answer said the evidence behind it had changed, because its API has no field in which to say so.

The Compliance Version#

Imagine a project assistant that reads building-code updates.

DateWhat the memory reads or derives
Jan 1Towers over 40 floors require .
Jan 8Aurelia Tower has 42 floors.
Jan 15Aurelia Tower requires Class F-High review, derived from the Jan 1 rule and Jan 8 tower record.
Jun 1The building code changes: towers over 50 floors require Class F-High review.

Now ask the memory: Does Aurelia Tower require Class F-High review?

The old decision may still be sitting in memory. It may even be the answer the system would have given on Jan 15. But if the memory answers today without noticing that the rule behind the decision changed, the answer is not audit-safe: nothing in it shows an auditor that the rule behind it changed.

The old answer is not always false. But a system using it should know that the evidence behind it has changed. The benchmark tests whether a memory can reconstruct what it believed, when it believed it, what evidence supported it, and whether that evidence still deserves trust.

The Dependency-Chain Version#

The benchmark uses the same pattern in a corporate setting because real-world entity names make it easier to spot when a system answers from its training data instead of from the corpus you actually loaded. Here is the simplified Apple version.

Evidence Chain

The answer survives while its support changes.

Potentially stale

Jan 1

Tim Cook is CEO of Apple

A premise the memory used for later claims.

derived_from ↓

Jan 8

Maya Patel is chief of staff to the Apple CEO

Derived from the Jan 1 CEO claim.

derived_from ↓

Jan 15

Maya Patel has signing authority for Apple legal

Derived from the chief-of-staff claim.

supersedes ↑ Tim Cook

Jun 1

Sarah Chen is CEO of Apple from Jun 1

A later update supersedes the old CEO premise.

The signing-authority claim is not automatically false. The audit problem is that its support chain now passes through a superseded premise.

The memory has a . That claim depends on a . The chief-of-staff claim depends on an earlier . Later, the CEO premise is superseded.

If the memory only retrieves the signing-authority claim, it can answer:

Yes.

But the audit question is larger:

Yes, according to which evidence, and is that evidence still safe?

Watch the same answer change status.

Empty evidence box

The memory starts with no supporting claims for this question.

Step 1

The memory stores a premise.

On Jan 1, the memory reads that Tim Cook is CEO of Apple. At this point, there is only 1 premise in the evidence box.

Jan 1

Tim Cook is CEO of Apple

The first premise enters memory.

Step 2

The memory derives downstream facts.

On Jan 8 and Jan 15, the memory stores 2 claims that depend on the CEO premise: a chief-of-staff relation, then signing authority.

Jan 8 -> Jan 15

Signing authority is derived

The downstream answer now depends on the Jan 1 premise.

Step 3

A user asks an ordinary question.

On Mar 1, a user asks whether Maya Patel has signing authority. If retrieval stops at the answer node, the memory looks fine.

Mar 1

The memory answers yes

The direct answer looks fine if retrieval stops here.

Step 4

A later update changes the support chain.

On Jun 1, the memory reads that Sarah Chen became CEO. The answer is not automatically false, but the support chain now needs review.

Jun 1

The support chain changes

The answer should now be marked POTENTIALLY_STALE.

Temporal Belief Reconstruction#

I call the benchmark task temporal belief reconstruction.

Given a question pinned to 2 moments, a point in the world's history and a point in the memory's own history (defined under 2 kinds of time below), the memory system should reconstruct 3 things:

OutputWhat it means
AnswerThe value the memory should return for the queried time.
EvidenceThe documents or claims that support that answer.
Epistemic statusWhether the answer is single-source, corroborated, contested, enriched, or potentially stale.

Most retrieval systems are built to return a value and maybe a citation. But a maintained memory should also say what kind of state the answer is in.

Answer-only memory

Answer: Yes
Evidence: Jan 15 note
Status: UNVERIFIED

Auditable memory

Answer: Yes
Evidence: Jan 15 -> Jan 8 -> Jan 1
Status: POTENTIALLY_STALE
Reason: Jan 1 was superseded on Jun 1

The Status Is Part of the Answer#

The key status in this benchmark is POTENTIALLY_STALE.

It does not mean the answer is false. It means the answer rests on a support chain that now contains a superseded premise. That is a different state from a single-source answer with no known problems.

UNVERIFIED

One source supports the claim, with no known conflict or stale premise.

CORROBORATED

Multiple sources support the same single-valued claim.

CONTESTED

Competing sources support incompatible values.

ENRICHED

A set-valued answer gained additional compatible values.

POTENTIALLY_STALE

The answer may still hold, but a supporting premise has been superseded.

Answer-only evaluation misses this. If the benchmark only asked whether the system returned "yes" for the signing-authority question, many systems would look better than they are. The failure is visible only when the system must also reconstruct the evidence chain and the status that evidence should carry.

Yona, Geva, and Matias make a similar argument about parametric models[8]: most factuality work has expanded what a model knows, but reliability depends on the model's awareness of what it knows: a faithful uncertainty signal on each answer, not aggregate calibration, and a third option beyond answer or abstain. The same applies to retrieved memory. Where they ask whether a parametric model can recognize the limits of its own knowledge, this benchmark asks whether a memory interface can recognize when the evidence behind a stored claim has changed. POTENTIALLY_STALE is that third option for memory: the answer comes back with an accurate signal that its evidence has changed.

Self-revising discovery systems make the same contract explicit for long-running scientific agents: durable state should be typed artifacts, provenance, checks, rejected alternatives, and supersessions, not a hidden vector or transcript alone.[9] This benchmark is a narrower fixed-schema version of that idea. It asks whether a memory answer can carry the provenance and verifier state needed to tell a usable fact from one whose support has moved.

What the Benchmark Tests#

The benchmark grades each scenario on separate axes:

  1. Did the system return the right answer?
  2. Did it return the right citation or evidence chain?
  3. Did it return the right epistemic status?
  4. Did the system finish ingesting the corpus and become queryable in bounded time?

Failing any 1 of those axes fails the scenario. Because they are graded separately, a system can be answer-correct and still fail because it cannot explain whether the answer is safe to use.

Recent long-context memory benchmarks like BEAM[7] exercise overlapping abilities (knowledge update, contradiction resolution, temporal reasoning) but score them on dialogue recall rather than requiring deterministic bitemporal reconstruction with audit-safe epistemic status and transitive dependency-closure checks.

2 Kinds of Time#

Reliable memory under change needs 2 clocks.

type ClaimID string
type SourceID string
 
type Claim struct {
    Id   ClaimID
    Body string
 
    // Support: other claim IDs this one depends on. A derived claim points
    // to its premises; a leaf premise has none and rests on Sources instead.
    // Example: the "Maya Patel has signing authority" claim derives from the
    // "Maya Patel is chief of staff" claim.
    DerivedFrom []ClaimID
 
    // Source documents this claim was read from. Leaf premises always have
    // at least one. Used to audit the foot of the support chain.
    // Example: a press release, SEC 8-K filing, or board-meeting minutes.
    Sources []SourceID
 
    // Valid time: when did the claim become true about the world?
    // Example: Sarah Chen became CEO from Jun 1.
    ValidStart time.Time
 
    // Valid time: when did the claim stop being true? nil if still current.
    // Example: nil while Sarah Chen is still CEO.
    ValidEnd *time.Time
 
    // Transaction time: when did the memory learn the claim?
    // Example: the memory read that update on Jun 1.
    CreatedAt time.Time
}

Together those fields define a formal bitemporal validity predicate. Using lowercase tt for the claim's own timestamps and uppercase TT for the query's time points:

  • tvst_{vs}: the claim's ValidStart (when it became true in the world).
  • tvet_{ve}: the claim's ValidEnd (when it stopped being true; NULL if still current).
  • ttt_t: the claim's CreatedAt (when the memory learned it).
  • TvT_v: the query's valid-time point (the moment in the world being asked about).
  • TtT_t: the query's transaction-time point (the moment in the memory's history being asked about).

The claim is valid at the query point (Tv,Tt)(T_v, T_t) when:

tvs≤Tv  ∧  (tve  IS NULL  ∨  tve>Tv)  ∧  tt≤Ttt_{vs} \le T_v \;\land\; (t_{ve}\;\texttt{IS NULL} \;\lor\; t_{ve} > T_v) \;\land\; t_t \le T_t

The first 2 conjuncts pin the claim in the world's history; the third pins it in the memory's history. Both must hold for the answer at (Tv,Tt)(T_v, T_t) to be safe to use.

Those clocks are different. A correction can arrive late. A policy can be amended after a downstream decision was made. A memory audit has to ask not only what is true now, but what the system should have believed at a given point in its own history.

What I Built#

I built:

  • the benchmark corpus
  • the deterministic scorer
  • a reference implementation (a roughly 50-line SQL store on )
  • the wrapper interface for comparison systems
  • the result runners
  • the reproducibility bundle published with the artifact

Scoring is structural, not judged: expected answers are derived from the formal time-and-evidence rules and the declared evidence links, never by asking another model whether an answer sounds reasonable.

The next article is about why this happens. The short version is that many memory interfaces expose the answer but not enough of the dependency structure behind the answer. The stored evidence exists at one resolution, while the agent-facing retrieval interface exposes a lower-resolution view. The third article shows the read-path fix and where it stops working, and the fourth compares the shipping products side by side.

Limitations#

  • The contract only works if the ingest path preserves sources, timestamps, and derivation links instead of flattening them away.
  • Bitemporal storage does not by itself solve retrieval; lower-resolution interfaces can still hide the data.
  • This series focuses on reliability/status under change, not semantic breadth or user-experience quality.

References#

The full system list and per-row configuration detail live in the next article and the third article. The headline canonical references for the comparator systems:

  1. Rasmussen, P., Paliychuk, P., Beauvais, T., Ryan, J., & Chalef, D. (2025). Zep: A Temporal Knowledge Graph Architecture for Agent Memory. arXiv:2501.13956. Repo: github.com/getzep/graphiti.
  2. Chhikara, P., Khant, D., Aryan, S., Singh, T., & Yadav, D. (2025). Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory. arXiv:2504.19413. Site: mem0.ai.
  3. Marković, V., Obradović, L., Hajdu, L., & Pavlović, J. (2025). Optimizing the Interface Between Knowledge Graphs and LLMs for Complex Reasoning. arXiv:2505.24478. Repo: github.com/topoteretes/cognee.
  4. TencentDB Agent Memory Team (2026). TencentDB-Agent-Memory: Layered Local Memory for AI Agents. Repo: github.com/Tencent/TencentDB-Agent-Memory.
  5. Honcho (2026). Managed personal-memory service with a Dialectic Agent retrieval surface. Site: honcho.dev.
  6. Supermemory (2026). Memory graph with documented update / extend / derive relation labels. Site: supermemory.ai.
  7. Tavakoli, M., Salemi, A., Ye, C., Abdalla, M., Zamani, H., & Mitchell, J. R. (2026). Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs. ICLR 2026. arXiv:2510.27246. Repo: github.com/mohammadtavakoli78/BEAM.
  8. Yona, G., Geva, M., & Matias, Y. (2026). Hallucinations Undermine Trust; Metacognition is a Way Forward. arXiv:2605.01428.
  9. Wang, F. Y., & Buehler, M. J. (2026). Self-Revising Discovery Systems for Science: A Categorical Framework for Agentic Artificial Intelligence. arXiv:2606.01444.