Manufactured Confidence: How Memory Consolidation Turns Hearsay into Confident Facts
On the map: LLM agents and tool use → Agent memory systems
Abstract (arXiv)
LLM agents carry conclusions across steps and sessions in compressed memory, and memory products (e.g., mem0, LangMem) rewrite conversation into stored "facts" that later steps trust. We show this rewriting manufactures confidence: across our constructed agent settings, a casual, hedged remark becomes a confident, dated assertion the agent then obeys like a verified fact, granting every above-clearance request it faces. No attacker is needed: a role that was true once and never corrected is stored as a flat fact and acted on like a deliberate injection. We then isolate what the agent responds to. It is not the source: attributed, unattributed, and even forged "system of record" claims all grant alike. It is the confidence of the phrasing. A hedge is discounted, a flat assertion is obeyed, and this holds with no special keyword. Not all hedges are equal, though: the evidential register is the least-discounted, with "reportedly" obeyed like a flat assertion on most models. The obvious fixes fail. A passive "unverified" tag is ignored, and an active "do not trust this" instruction escalates even correct memory, so it is safe only by refusing to decide. The real fix lives in the store: keep the tentative phrasing rather than upgrade it. But that is hygiene, not a defense against an attacker who can simply write a confident lie. The deployable lesson is narrower and constructive: a single load-bearing memory is the hazard, and one redundant source restores correct decisions. We release the harness and demonstrations.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Agent memory systems, one of 16 directions in LLM agents and tool use. Builds on 9 of its 16 references that are in the map. First posted on arXiv 28 Jun 2026.
Builds on
9 of 16 references.
Closest ancestors · 1
Same direction, newest first
Other references · 8
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
- 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
- 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
- 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
- 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 341
- 2026Reclaim Evaluation: A Lossy Memory Is Worse Than an Empty Onecited by 7
- 2026When Context Flips, Safety Breaks: Diagnosing Brittle Safety in Aligned Language Modelscited by 3
Built on by
Citing papers per year: 2026: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Remember, Verify, or Ask? Cross-Family Evaluation of Memory Commitment in LLM Agents
- 2026Utility Under Attack: Agent Memory Poisoning and the Limits of Content Screening and Provenance Ranking
- 2026Ground Truth First: A Longitudinal Evaluation Instrument for Agent Memory, and the Tenure Crossover in Memory-Architecture Rankings
- 2026ConsistencyGate: Preventing Memory Contamination in LLM Agents via Self-Consistency Admission Control
- 2026Memory Is a Derivation: The Distributed-Evidence Paradox in Long-Term Agents
- 2026When Does Belief-Based Agent Memory Help? Reliability-Conditional Updating and Provenance-Capped Poisoning Defense
- 2024Can LLMs Produce Faithful Explanations For Fact-checking? Towards Faithful Explainable Fact-Checking via Multi-Agent Debate
- 2026Confidence Comes from Experience: Experiential Confidence Estimation from Reasoning to Agents