# On Safety Risks in Experience-Driven Self-Evolving Agents

arXiv: https://arxiv.org/abs/2604.16968 · HTML: https://mlsanity.com/paper/2604.16968/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 31 of its 54 references that are in the map.
- First posted on arXiv 18 Apr 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)
- Synthetic data and self-improvement for LLMs / [Self-evolving LLM agents](https://mlsanity.com/synth/self-evolving-llm-agents/)

## Builds on (most cited first, 30 of 31)

- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [ExpeL: LLM Agents Are Experiential Learners](https://mlsanity.com/paper/2308.10144/) (2023)
- [Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models](https://mlsanity.com/paper/2510.04618/) (2025)
- [Agent-SafetyBench: Evaluating the Safety of LLM Agents](https://mlsanity.com/paper/2412.14470/) (2024)
- [Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning](https://mlsanity.com/paper/2305.14160/) (2023)
- [A Comprehensive Survey of Self-Evolving AI Agents: A New Paradigm Bridging Foundation Models and Lifelong Agentic Systems](https://mlsanity.com/paper/2508.07407/) (2025)
- [Agentic Misalignment: How LLMs Could Be Insider Threats](https://mlsanity.com/paper/2510.05179/) (2025)
- [Context Length Alone Hurts LLM Performance Despite Perfect Retrieval](https://mlsanity.com/paper/2510.05381/) (2025)
- [SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents](https://mlsanity.com/paper/2412.13178/) (2024)
- [Lifelong Learning of Large Language Model based Agents: A Roadmap](https://mlsanity.com/paper/2501.07278/) (2025)
- [Open-Endedness is Essential for Artificial Superhuman Intelligence](https://mlsanity.com/paper/2406.04268/) (2024)
- [AutoManual: Constructing Instruction Manuals by LLM Agents via Interactive Environmental Learning](https://mlsanity.com/paper/2405.16247/) (2024)
- [Contextual Experience Replay for Self-Improvement of Language Agents](https://mlsanity.com/paper/2506.06698/) (2025)
- [Consent in Crisis: The Rapid Decline of the AI Data Commons](https://mlsanity.com/paper/2407.14933/) (2024)
- [Building Self-Evolving Agents via Experience-Driven Lifelong Learning: A Framework and Benchmark](https://mlsanity.com/paper/2508.19005/) (2025)
- [MoT: Memory-of-Thought Enables ChatGPT to Self-Improve](https://mlsanity.com/paper/2305.05181/) (2023)
- [Failures Pave the Way: Enhancing Large Language Models through Tuning-free Rule Accumulation](https://mlsanity.com/paper/2310.15746/) (2023)
- [A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents](https://mlsanity.com/paper/2506.23844/) (2025)
- [Accumulating Context Changes the Beliefs of Language Models](https://mlsanity.com/paper/2511.01805/) (2025)
- [Coarse-to-Fine Grounded Memory for LLM Agent Planning](https://mlsanity.com/paper/2508.15305/) (2025)
- [The Future of Continual Learning in the Era of Foundation Models: Three Key Directions](https://mlsanity.com/paper/2506.03320/) (2025)
- [Open Questions in Creating Safe Open-ended AI: Tensions Between Control and Creativity](https://mlsanity.com/paper/2006.07495/) (2020)
- [Episodic memory in AI agents poses risks that should be studied and mitigated](https://mlsanity.com/paper/2501.11739/) (2025)
- [AI & Human Co-Improvement for Safer Co-Superintelligence](https://mlsanity.com/paper/2512.05356/) (2025)
- [Alignment Tipping Process: How Self-Evolution Pushes LLM Agents Off the Rails](https://mlsanity.com/paper/2510.04860/) (2025)
- [Large Language Model Agents Are Not Always Faithful Self-Evolvers](https://mlsanity.com/paper/2601.22436/) (2026)
- [The PacifAIst Benchmark:Would an Artificial Intelligence Choose to Sacrifice Itself for Human Safety?](https://mlsanity.com/paper/2508.09762/) (2025)
- [Safety Must Precede the Deployment of Open-Ended AI](https://mlsanity.com/paper/2502.04512/) (2025)

## Built on by (most cited first, 14 of 14)

- [On-Policy Self-Evolution via Failure Trajectories for Agentic Safety Alignment](https://mlsanity.com/paper/2605.11882/) (2026)
- [Are You Still the Agent I Authorized? Earned Authority under a Fixed Ceiling for Evolving Agents](https://mlsanity.com/paper/2607.23586/) (2026)
- [Auditing Harness Tampering in Self-Improving Agents](https://mlsanity.com/paper/2609.00069/) (2026)
- [Practice Makes Unsafe: Skill Misevolution in Self-Improving LLM Agents](https://mlsanity.com/paper/2608.12851/) (2026)
- [EVOMAL: Self-Poisoning in Self-Evolving Coding Agents](https://mlsanity.com/paper/2608.25776/) (2026)
- [OmniHarness: Harnessing Generalizable Visual Generation via Symbolic Policy Learning](https://mlsanity.com/paper/2609.16057/) (2026)
- [Rethinking Experience Utilization in Self-Evolving Language Model Agents](https://mlsanity.com/paper/2605.07164/) (2026)
- [ANCHOR: An External LLM-Driven Supervisory Module Facilitating Healthy Evolution in Self-Evolving Systems](https://mlsanity.com/paper/2606.06114/) (2026)
- [Reflections on Trusting Trust, Revisited: Contaminating Self-Modifying AI Coding Agents with Poisoned Benchmarks](https://mlsanity.com/paper/2609.17817/) (2026)
- [Evolutionary Safety of Recursive Self-Improving AI: Taxonomy, Risk Discovery, and Evaluation](https://mlsanity.com/paper/2609.31186/) (2026)
- [Compositional Safety Failures in Harness Evolution: Identification and Runtime Monitoring](https://mlsanity.com/paper/2609.33123/) (2026)
- [When Consent Outlives Context: Residual Authority Replay in Long-Lived Agents](https://mlsanity.com/paper/2609.33910/) (2026)
- [VEPA: Verified Evolving Formal Policy for Lifelong LLM Agent Guardrails](https://mlsanity.com/paper/2610.12813/) (2026)
- [EvolvAttack: Backdooring self Evolving VLM Agents](https://mlsanity.com/paper/2610.15244/) (2026)

## Abstract (arXiv)

Experience-driven self-evolution has emerged as a promising paradigm for improving the autonomy of large language model agents, yet its reliance on self-curated experience introduces underexplored safety risks. In this study, we investigate how experience accumulation and utilization in self-evolving agents affect safety performance across web-based and embodied environments. Notably, experience gathered solely from benign tasks can still compromise safety in high-risk scenarios. Further analysis attributes this degradation to the execution-oriented nature of accumulated experience, which reinforces agents' tendency to act rather than refuse. In more realistic settings where agents encounter both benign and harmful tasks, refusal-related experience mitigates safety decline but induces over-refusal, revealing a fundamental safety-utility trade-off. Overall, our findings expose inherent limitations of current self-evolving agents and call for more principled strategies to ensure safe and reliable adaptation.