# Rethinking Experience Utilization in Self-Evolving Language Model Agents

arXiv: https://arxiv.org/abs/2605.07164 · HTML: https://mlsanity.com/paper/2605.07164/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 23 of its 57 references that are in the map.
- First posted on arXiv 8 May 2026.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)
- Synthetic data and self-improvement for LLMs / [Self-evolving LLM agents](https://mlsanity.com/synth/self-evolving-llm-agents/)

## Builds on (most cited first, 23 of 23)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [OpenAI GPT-5 System Card](https://mlsanity.com/paper/2601.03267/) (2025)
- [Group-in-Group Policy Optimization for LLM Agent Training](https://mlsanity.com/paper/2505.10978/) (2025)
- [Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models](https://mlsanity.com/paper/2510.04618/) (2025)
- [ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory](https://mlsanity.com/paper/2509.25140/) (2025)
- [A Comprehensive Survey of Self-Evolving AI Agents: A New Paradigm Bridging Foundation Models and Lifelong Agentic Systems](https://mlsanity.com/paper/2508.07407/) (2025)
- [A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence](https://mlsanity.com/paper/2507.21046/) (2025)
- [Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory](https://mlsanity.com/paper/2504.07952/) (2025)
- [EvolveR: Self-Evolving LLM Agents through an Experience-Driven Lifecycle](https://mlsanity.com/paper/2510.16079/) (2025)
- [Memento: Fine-tuning LLM Agents without Fine-tuning LLMs](https://mlsanity.com/paper/2508.16153/) (2025)
- [AgentEvolver: Towards Efficient Self-Evolving Agent System](https://mlsanity.com/paper/2511.10395/) (2025)
- [Building Self-Evolving Agents via Experience-Driven Lifelong Learning: A Framework and Benchmark](https://mlsanity.com/paper/2508.19005/) (2025)
- [ExpSeek: Self-Triggered Experience Seeking for Web Agents](https://mlsanity.com/paper/2601.08605/) (2026)
- [Learning to Continually Learn via Meta-learning Agentic Memory Designs](https://mlsanity.com/paper/2602.07755/) (2026)
- [The Future of Continual Learning in the Era of Foundation Models: Three Key Directions](https://mlsanity.com/paper/2506.03320/) (2025)
- [Position: Agentic Evolution is the Path to Evolving LLMs](https://mlsanity.com/paper/2602.00359/) (2026)
- [On Safety Risks in Experience-Driven Self-Evolving Agents](https://mlsanity.com/paper/2604.16968/) (2026)
- [M^star: Every Task Deserves Its Own Memory Harness](https://mlsanity.com/paper/2604.11811/) (2026)
- [Large Language Model Agents Are Not Always Faithful Self-Evolvers](https://mlsanity.com/paper/2601.22436/) (2026)
- [Ask Only When Needed: Proactive Retrieval from Memory and Skills for Experience-Driven Lifelong Agents](https://mlsanity.com/paper/2604.20572/) (2026)
- [On Improvisation and Open-Endedness: Insights for Experiential AI](https://mlsanity.com/paper/2511.00529/) (2025)

## Built on by (most cited first, 1 of 1)

- [DeltaReplay: Task-Relative Memory Reuse for Mobile GUI Agents](https://mlsanity.com/paper/2610.11707/) (2026)

## Abstract (arXiv)

Self-evolving agents improve by accumulating and reusing experience from past interactions. Existing work has largely focused on how experience is constructed, represented, and updated, while paying less attention to how experience should be used during runtime decision-making. As a result, most agents rely on rigid usage strategies, either injecting experience once at initialization or at every step, without considering whether it is needed for the current decision. This paper studies experience utilization as a critical design dimension of self-evolving agents. We ask whether agents benefit from interweaving experience use with decision-making, so that experience is invoked only when additional guidance is needed. To examine this question, we introduce ExpWeaver, a lightweight instantiation that leaves experience construction unchanged and modifies only runtime utilization by exposing experience as an optional resource during reasoning. Across four representative frameworks, seven LLM backbones, and three types of environments, ExpWeaver consistently achieves the best performance among different utilization strategies. Reinforcement learning experiments further show that this behavior can be amplified through training. Usage-pattern, causal ablation, and entropy-based analyses reveal that ExpWeaver enables agents to invoke experience selectively, at beneficial decision points, and under higher reasoning uncertainty. Overall, our findings call for a shift from merely studying what experience to store toward understanding how and when experience should enter decision-making.