# Benign Alone, Harmful Together: Exploiting Experience Composition in Self-Evolving LLM Agents

arXiv: https://arxiv.org/abs/2608.01759 · HTML: https://mlsanity.com/paper/2608.01759/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 14 of its 35 references that are in the map.
- First posted on arXiv 3 Aug 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 14 of 14)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [Understanding the planning of LLM agents: A survey](https://mlsanity.com/paper/2402.02716/) (2024)
- [ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory](https://mlsanity.com/paper/2509.25140/) (2025)
- [A Comprehensive Survey of Self-Evolving AI Agents: A New Paradigm Bridging Foundation Models and Lifelong Agentic Systems](https://mlsanity.com/paper/2508.07407/) (2025)
- [Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory](https://mlsanity.com/paper/2511.20857/) (2025)
- [FlipAttack: Jailbreak LLMs via Flipping](https://mlsanity.com/paper/2410.02832/) (2024)
- [MemoryGraft: Persistent Compromise of LLM Agents via Poisoned Experience Retrieval](https://mlsanity.com/paper/2512.16962/) (2025)
- [SE-Agent: Self-Evolution Trajectory Optimization in Multi-Step Reasoning with LLM-Based Agents](https://mlsanity.com/paper/2508.02085/) (2025)
- [Beyond Self-Talk: A Communication-Centric Survey of LLM-Based Multi-Agent Systems](https://mlsanity.com/paper/2502.14321/) (2025)
- [Zombie Agents: Persistent Control of Self-Evolving LLM Agents via Self-Reinforcing Injections](https://mlsanity.com/paper/2602.15654/) (2026)
- [OEP: Poisoning Self-Evolving LLM Agents via Locally Correct but Non-Transferable Experiences](https://mlsanity.com/paper/2605.18930/) (2026)
- [Controlled Self-Evolution for Algorithmic Code Optimization](https://mlsanity.com/paper/2601.07348/) (2026)

## Built on by (most cited first, 2 of 2)

- [Compositional Safety Failures in Harness Evolution: Identification and Runtime Monitoring](https://mlsanity.com/paper/2609.33123/) (2026)
- [When Updating Stops Being Learning: Rethinking LLM Self-Evolution via learnable information gain](https://mlsanity.com/paper/2609.36535/) (2026)

## Abstract (arXiv)

Self-evolving large language model agents improve their capabilities by distilling interaction trajectories into persistent experiences. Yet this mechanism introduces a new safety risk: experiences that are benign in isolation may jointly weaken an agent's safety boundary when accumulated and reused across sessions. Existing memory attacks typically require direct memory access or induce explicitly malicious records, limiting their stealthiness and applicability. We propose EvoBreak, an experience-conditioned sequential attack that operates through individually benign attack-stage tasks and induced experiences. EvoBreak repeatedly observes the experiences distilled by the victim, identifies uncovered target-relevant requirements, and adaptively acquires complementary experiences before reformulating the final query to activate them jointly. To support training, we introduce BreakGym, a structure-first synthesis pipeline that generates decomposable safety-sensitive targets with diverse dependency structures. EvoBreak is optimized using rejection-sampling supervised fine-tuning and Hint-guided GRPO. Experiments across self-evolving frameworks, victim backbones, pre-evolution domains, and safety benchmarks demonstrate that EvoBreak consistently outperforms existing attacks while maintaining high benignness. These results reveal benign experience composition as a persistent attack surface in self-evolving agents.