mlsanityData to 14 Oct 2026

Your Agent May Misevolve: Emergent Risks in Self-evolving LLM Agents

arXiv 2509.26354 ↗First posted 30 Sep 2025cs.AI cs.CL cs.LGcited by 62

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

Advances in Large Language Models (LLMs) have enabled a new class of self-evolving agents that autonomously improve through interaction with the environment, demonstrating strong capabilities. However, self-evolution also introduces novel risks overlooked by current safety research. In this work, we study the case where an agent's self-evolution deviates in unintended ways, leading to undesirable or even harmful outcomes. We refer to this as Misevolution. To provide a systematic investigation, we evaluate misevolution along four key evolutionary pathways: model, memory, tool, and workflow. Our empirical findings reveal that misevolution is a widespread risk, affecting agents built even on top-tier LLMs (e.g., Gemini-2.5-Pro). Different emergent risks are observed in the self-evolutionary process, such as the degradation of safety alignment after memory accumulation, or the unintended introduction of vulnerabilities in tool creation and reuse. To our knowledge, this is the first study to systematically conceptualize misevolution and provide empirical evidence of its occurrence, highlighting an urgent need for new safety paradigms for self-evolving agents. Finally, we discuss potential mitigation strategies to inspire further research on building safer and more trustworthy self-evolving agents. Our code and data are available at https://github.com/ShaoShuai0605/Misevolution . Warning: this paper includes examples that may be offensive or harmful in nature.

Key facts. Cited by 62 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 46 of its 81 references that are in the map. First posted on arXiv 30 Sep 2025.

Builds on

46 of 81 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025A-MemGuard: A Proactive Defense Framework for LLM-Based Agent Memory29 Sep
  2. 2025MCP-Guard: A Multi-Stage Defense-in-Depth Framework for Securing Model Context Protocol in Agentic AI14 Aug
  3. 2025RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agents31 May
  4. 2025Dynamic Risk Assessments for Offensive Cybersecurity Agents23 May
  5. 2025AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agents12 Mar
  6. 2025SafeArena: Evaluating the Safety of Autonomous Web Agents6 Mar
  7. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agents19 Dec
  8. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies28 Jul
  9. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training10 Jan

Other references · 37

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2024GPT-4o System Cardcited by 4,588
  6. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  8. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  9. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  10. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  11. 2025AlphaEvolve: A coding agent for scientific and algorithmic discoverycited by 867
  12. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 610
25 more
  1. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 472
  2. 2025Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agentscited by 356
  3. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 344
  4. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 333
  5. 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 286
  6. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
  7. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 228
  8. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 228
  9. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 192
  10. 2025A Comprehensive Survey of Self-Evolving AI Agents: A New Paradigm Bridging Foundation Models and Lifelong Agentic Systemscited by 183
  11. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 170
  12. 2025Multi-Agent Risks from Advanced AIcited by 163
  13. 2025Memento: Fine-tuning LLM Agents without Fine-tuning LLMscited by 127
  14. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  15. 2025Alita: Generalist Agent Enabling Scalable Agentic Reasoning with Minimal Predefinition and Maximal Self-Evolutioncited by 116
  16. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 90
  17. 2025AgentNet: Decentralized Evolutionary Coordination for LLM-based Multi-Agent Systemscited by 90
  18. 2025PyVision: Agentic Vision with Dynamic Toolingcited by 79
  19. 2025SE-Agent: Self-Evolution Trajectory Optimization in Multi-Step Reasoning with LLM-Based Agentscited by 69
  20. 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 69
  21. 2025Self-Challenging Language Model Agentscited by 59
  22. 2024From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactionscited by 46
  23. 2025EvoAgentX: An Automated Framework for Evolving Agentic Workflowscited by 44
  24. 2025Data Poisoning in Deep Learning: A Surveycited by 20
  25. 2025Advanced Tool Learning and Selection System (ATLASS): A Closed-Loop Framework Using LLMcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 60.

62 papers.

Sort:

By area:

  1. 2026Zombie Agents: Persistent Control of Self-Evolving LLM Agents via Self-Reinforcing Injectionscited by 48
  2. 2026Your Agent, Their Asset: A Real-World Safety Analysis of OpenClawcited by 20
  3. 2025Are Your Agents Upward Deceivers?cited by 16
  4. 2026MMSkills: Towards Multimodal Skills for General Visual Agentscited by 16
  5. 2026OEP: Poisoning Self-Evolving LLM Agents via Locally Correct but Non-Transferable Experiencescited by 14
  6. 2026The Why Behind the Action: Unveiling Internal Drivers via Agentic Attributioncited by 11
  7. 2026Belief Memory: Agent Memory Under Partial Observabilitycited by 11
  8. 2026AgentForesight: Online Auditing for Early Failure Prediction in Multi-Agent Systemscited by 11
  9. 2026TAME: A Trustworthy Test-Time Evolution of Agent Memory with Systematic Benchmarkingcited by 10
  10. 2026Evolving-RL: End-to-End Optimization of Experience-Driven Self-Evolving Capability within Agentscited by 9
  11. 2026Recursive Experiential-Working Memory Evolution for Long-Horizon Agent Harnessescited by 9
  12. 2026Risky-Bench: Probing Agentic Safety Risks under Real-World Deploymentcited by 8
38 more
  1. 2026On-Policy Self-Evolution via Failure Trajectories for Agentic Safety Alignmentcited by 8
  2. 2026Safety in Embodied AI: A Survey of Risks, Attacks, and Defensescited by 7
  3. 2026Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influencecited by 6
  4. 2026Do Self-Evolving Agents Forget? Capability Degradation and Preservation in Lifelong LLM Agent Adaptationcited by 6
  5. 2026EvoFSM: Controllable Self-Evolution for Deep Research with Finite State Machinescited by 5
  6. 2026MonoScale: Scaling Multi-Agent System with Monotonic Improvementcited by 5
  7. 2026Are You Still the Agent I Authorized? Earned Authority under a Fixed Ceiling for Evolving Agentscited by 5
  8. 2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 4
  9. 2026SafeHarbor: Defining Precise Decision Boundaries via Hierarchical Memory-Augmented Guardrail for LLM Agent Safetycited by 4
  10. 2026AgentStream: How Well Do Self-Evolving LLM Agents Perform Under Streaming Tasks?cited by 4
  11. 2026Auditing Harness Tampering in Self-Improving Agentscited by 4
  12. 2026AIvilization v0: Toward Large-Scale Artificial Social Simulation with a Unified Agent Architecture and Adaptive Agent Profilescited by 3
  13. 2026What Do Evolutionary Coding Agents Evolve?cited by 3
  14. 2026PACE: Two-Timescale Self-Evolution for Small Language Model Agentscited by 3
  15. 2026Practice Makes Unsafe: Skill Misevolution in Self-Improving LLM Agentscited by 3
  16. 2026EVOMAL: Self-Poisoning in Self-Evolving Coding Agentscited by 3
  17. 2026Ecdysis: Efficient and Effective Training of Runtime Harnesses for LLM Agentscited by 3
  18. 2026Agentic AI for Remote Sensing: Technical Challenges and Research Directionscited by 2
  19. 2026Towards Direct Evaluation of Harness Optimizers via Priority Rankingcited by 2
  20. 2026Phantom Gains: Auditing Self-Improvement Against a Measured Nullcited by 2
  21. 2026MetaRSI / RSI2: A Meta-Recursive Self-Improving System for Recursive Self-Improving Systems Themselvescited by 2
  22. 2026VirtualCrime: Evaluating the Criminal Potential and Agentic Behaviors of Large Language Models via Sandbox Simulationcited by 1
  23. 2026AI Security in the Foundation Model Era: A Comprehensive Survey from a Unified Perspectivecited by 1
  24. 2026Banana100: Breaking NR-IQA Metrics by 100 Iterative Image Replications with Nano Banana Procited by 1
  25. 2026MemEvoBench: Benchmarking Safety Risks from Memory Misevolution in LLM Agentscited by 1
  26. 2026ANCHOR: An External LLM-Driven Supervisory Module Facilitating Healthy Evolution in Self-Evolving Systemscited by 1
  27. 2026Open Problems in AI Incident Governancecited by 1
  28. 2026Agents in the Large: Perception-Centered Architecture for Persistent Agentscited by 1
  29. 2026When Is Enough Enough in Self-Evolving LLM Systems?cited by 1
  30. 2025Systematizing LLM Persona Design: A Four-Quadrant Technical Taxonomy for AI Companion Applicationscited by 0
  31. 2026ANNEAL: Adapting LLM Agents via Governed Symbolic Patch Learningcited by 0
  32. 2026Physiological Noise Augmentation Improves Non-Invasive Brain-to-Speechcited by 0
  33. 2026LOGOS: A Living Logic for AI Agent Teams That Evolve With Humanscited by 0
  34. 2026Safe Evolution with Circuit Anchorscited by 0
  35. 2026Auditing Self-Evolution in Financial Agents: Capability Gains, Security Drift, and Execution-Interface Mismatchcited by 0
  36. 2026Forgotten in Weights, Recovered by Tools: Agentic Tool Unlearning for LLM Agentscited by 0
  37. 2026Reflections on Trusting Trust, Revisited: Contaminating Self-Modifying AI Coding Agents with Poisoned Benchmarkscited by 0
  38. 2026Evolutionary Safety of Recursive Self-Improving AI: Taxonomy, Risk Discovery, and Evaluationcited by 0

50 most cited of 62 All 62 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Safety in Self-Evolving LLM Agent Systems: Threats, Amplification, and Case Studies
  2. 2026On Safety Risks in Experience-Driven Self-Evolving Agents
  3. 2026The Devil Behind Moltbook: Anthropic Safety is Always Vanishing in Self-Evolving AI Societies
  4. 2026SHE: Trajectory-driven Safety Harness Evolution for LLM Agents
  5. 2025Alignment Tipping Process: How Self-Evolution Pushes LLM Agents Off the Rails
  6. 2025AgentMisalignment: Measuring the Propensity for Misaligned Behaviour in LLM-Based Agents
  7. 2026SafeEvolve: Harness-Policy Co-Evolution from Agent Experience for Safety Alignment
  8. 2026Beyond Individual Intelligence: Surveying Collaboration, Failure Attribution, and Self-Evolution in LLM-based Multi-Agent Systems