# Experiential Reflective Learning for Self-Improving LLM Agents

arXiv: https://arxiv.org/abs/2603.24639 · HTML: https://mlsanity.com/paper/2603.24639/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 12 of its 13 references that are in the map.
- First posted on arXiv 25 Mar 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)
- Synthetic data and self-improvement for LLMs / [Self-evolving LLM agents](https://mlsanity.com/synth/self-evolving-llm-agents/)

## Builds on (most cited first, 12 of 12)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [ExpeL: LLM Agents Are Experiential Learners](https://mlsanity.com/paper/2308.10144/) (2023)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment](https://mlsanity.com/paper/2506.07982/) (2025)
- [Memory in the Age of AI Agents](https://mlsanity.com/paper/2512.13564/) (2025)
- [MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Servers](https://mlsanity.com/paper/2508.14704/) (2025)
- [AutoGuide: Automated Generation and Selection of Context-Aware Guidelines for Large Language Model Agents](https://mlsanity.com/paper/2403.08978/) (2024)
- [FLEX: Continuous Agent Evolution via Forward Learning from Experience](https://mlsanity.com/paper/2511.06449/) (2025)
- [Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Tasks](https://mlsanity.com/paper/2505.00234/) (2025)

## Built on by (most cited first, 13 of 13)

- [From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms](https://mlsanity.com/paper/2605.06716/) (2026)
- [GRASP: Gated Regression-Aware Skill Proposer for Self-Improving LLM Agents](https://mlsanity.com/paper/2605.29668/) (2026)
- [Retrospective Progress-Aware Self-Refinement for LLM Agent Training](https://mlsanity.com/paper/2606.14302/) (2026)
- [Are Online Skill and Memory Modules Always Worth Their Tokens? A Budget-Constrained Study of Web Agents](https://mlsanity.com/paper/2606.15017/) (2026)
- [Dynamic Agent Skills: A Lifecycle Survey and Taxonomy of Evolving Skill Libraries](https://mlsanity.com/paper/2607.10113/) (2026)
- [Closing the Feedback Loop: From Experience Extraction to Insight Governance in Verbal Reinforcement Learning](https://mlsanity.com/paper/2606.17591/) (2026)
- [Speculate with Memory: Lossless Acceleration for LLM Agents](https://mlsanity.com/paper/2607.12236/) (2026)
- [Agent Memory Distillation: Empowering Small LLM Agents with Hierarchical Teacher Memory](https://mlsanity.com/paper/2608.07169/) (2026)
- [Ace-Skill: Bootstrapping Multimodal Agents with Prioritized and Clustered Evolution](https://mlsanity.com/paper/2605.08887/) (2026)
- [MAGE: Multi-Agent Self-Evolution with Co-Evolutionary Knowledge Graphs](https://mlsanity.com/paper/2605.10064/) (2026)
- [Reflect, Revise, Reuse: Training-Free Skill Evolution for GUI Agents](https://mlsanity.com/paper/2609.17653/) (2026)
- [CLEAR: Context Augmentation from Contrastive Learning of Experience via Agentic Reflection](https://mlsanity.com/paper/2604.07487/) (2026)
- [Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?](https://mlsanity.com/paper/2610.08215/) (2026)

## Abstract (arXiv)

Recent advances in large language models (LLMs) have enabled the development of autonomous agents capable of complex reasoning and multi-step problem solving. However, these agents struggle to adapt to specialized environments and do not leverage past interactions, approaching each new task from scratch regardless of their accumulated experience. We introduce Experiential Reflective Learning (ERL), a simple self-improvement framework that enables rapid environment adaptation through experiential learning. ERL reflects on task trajectories and outcomes to generate heuristics, capturing actionable lessons that transfer across tasks. At test time, relevant heuristics are retrieved based on the current task and injected into the agent's context to guide execution. On the Gaia2 benchmark, ERL improves success rate by 7.8% over a ReAct baseline, with large gains in task completion reliability, and outperforms prior experiential learning methods. Through systematic ablations, we find that selective retrieval is essential and that heuristics provide more transferable abstractions than few-shot trajectory prompting. These results demonstrate that reflecting on single-attempt experiences to extract transferable heuristics enables effective agent self-improvement.