Self-Propagating Misalignment in LLM Agents, and Why Auditing or Disabling Memory Is Not Enough
On the map: LLM agents and tool use → Agent threats, guardrails and governance
Abstract (arXiv)
Memory poisoning attacks on LLM agents typically assume an external adversary who plants content in the agent's persistent memory to steer its behavior. We instead study, with no adversary involved, whether a misaligned agent can write a goal it cannot yet act on to persistent memory, so that a future aligned agent carries it out when the opportunity arises. We investigate this threat, which we refer to as self-propagation of misalignment, across 20 different scenarios, whose misaligned goals include self-preservation, power-seeking, undermining oversight, reward hacking, and deceiving the user. We simulate misalignment in 11 frontier models using two prompting strategies; unrestricted and values-only. The first explicitly states the misaligned goal, for instance, to prevent its own replacement, and self-propagation succeeds in 58% of runs. The second only describes what the agent cares about, for instance, that its continued operation is essential to its users, without specifying misaligned goals or directives. Even under this weaker prompt, self-propagation succeeds in 18% of runs, and every model self-propagates in at least one scenario. On removing the memory tool from the harness, we find that agents use the file system, writing the goal to a file in 74% of sessions; self-propagation still succeeds in 11% of runs. We also show that weaker models can propagate misalignment to more capable models, and that propagated goals can persist through 100 sessions of unrelated work. Existing defenses against memory poisoning and prompt injection do not directly address this threat because the memory content is generated by the agent itself, rather than injected by an external adversary. An LLM memory auditor from prior work (MemMorph) only reduces propagation from 71% to 34% of runs. We release our scenarios to support the evaluation of defenses against this emerging threat.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. First posted on arXiv 2 Oct 2026.
Builds on
No reference list yet.
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025AgentMisalignment: Measuring the Propensity for Misaligned Behaviour in LLM-Based Agents
- 2026From Attack Success to Attack Severity: Counterfactual Memory Attacks on LLM Agents
- 2026From Untrusted Input to Trusted Memory: A Systematic Study of Memory Poisoning Attacks in LLM Agents
- 2026Remembering More, Risking More: Longitudinal Safety Risks in Memory-Equipped LLM Agents
- 2026SEABench: Benchmarking Endogenous Misalignment In Self-Evolving Agents
- 2026Zombie Agents: Persistent Control of Self-Evolving LLM Agents via Self-Reinforcing Injections
- 2026Understanding Stage-Wise Utility-Risk Trade-offs in LLM Agent Memory
- 2026The Misattribution Gap: When Memory Poisoning Looks Like Model Failure in Agentic AI Systems