# SAMULE: Self-Learning Agents Enhanced by Multi-level Reflection

arXiv: https://arxiv.org/abs/2509.20562 · HTML: https://mlsanity.com/paper/2509.20562/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 21 of its 36 references that are in the map.
- First posted on arXiv 24 Sep 2025.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 21 of 21)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [ExpeL: LLM Agents Are Experiential Learners](https://mlsanity.com/paper/2308.10144/) (2023)
- [TravelPlanner: A Benchmark for Real-World Planning with Language Agents](https://mlsanity.com/paper/2402.01622/) (2024)
- [WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning](https://mlsanity.com/paper/2411.02337/) (2024)
- [NATURAL PLAN: Benchmarking LLMs on Natural Language Planning](https://mlsanity.com/paper/2406.04520/) (2024)
- [Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization](https://mlsanity.com/paper/2308.02151/) (2023)
- [Self-Contrast: Better Reflection Through Inconsistent Solving Perspectives](https://mlsanity.com/paper/2401.02009/) (2024)
- [Evolving Deeper LLM Thinking](https://mlsanity.com/paper/2501.09891/) (2025)
- [Experiential Co-Learning of Software-Developing Agents](https://mlsanity.com/paper/2312.17025/) (2023)
- [Learning From Failure: Integrating Negative Examples when Fine-tuning Large Language Models as Agents](https://mlsanity.com/paper/2402.11651/) (2024)
- [TReMu: Towards Neuro-Symbolic Temporal Reasoning for LLM-Agents with Memory in Multi-Session Dialogues](https://mlsanity.com/paper/2502.01630/) (2025)
- [Testing and Understanding Erroneous Planning in LLM Agents through Synthesized User Inputs](https://mlsanity.com/paper/2404.17833/) (2024)
- [MetaReflection: Learning Instructions for Language Agents using Past Reflections](https://mlsanity.com/paper/2405.13009/) (2024)
- [Iteration of Thought: Leveraging Inner Dialogue for Autonomous Large Language Model Reasoning](https://mlsanity.com/paper/2409.12618/) (2024)
- [Language Guided Exploration for RL Agents in Text Environments](https://mlsanity.com/paper/2403.03141/) (2024)
- [Self-Evolving GPT: A Lifelong Autonomous Experiential Learner](https://mlsanity.com/paper/2407.08937/) (2024)
- [Optimizing LLM-Based Multi-Agent System with Textual Feedback: A Case Study on Software Development](https://mlsanity.com/paper/2505.16086/) (2025)
- [Contrastive Retrospection: honing in on critical steps for rapid learning and generalization in RL](https://mlsanity.com/paper/2210.05845/) (2022)
- [Improving Retrospective Language Agents via Joint Policy Gradient Optimization](https://mlsanity.com/paper/2503.01490/) (2025)

## Built on by (most cited first, 15 of 15)

- [SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning](https://mlsanity.com/paper/2607.14777/) (2026)
- [Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?](https://mlsanity.com/paper/2607.09996/) (2026)
- [Retrospective Progress-Aware Self-Refinement for LLM Agent Training](https://mlsanity.com/paper/2606.14302/) (2026)
- [Deep Tabular Research via Continual Experience-Driven Execution](https://mlsanity.com/paper/2603.09151/) (2026)
- [From Memory to Skills: Evidence-Grounded Co-Evolution Governance for Long-Horizon LLM Agents](https://mlsanity.com/paper/2607.16621/) (2026)
- [ReEXplore: Improving MLLMs for Embodied Exploration with Contextualized Retrospective Experience Replay](https://mlsanity.com/paper/2511.19033/) (2025)
- [PreFlect: From Retrospective to Prospective Reflection in Large Language Model Agents](https://mlsanity.com/paper/2602.07187/) (2026)
- [BenchTrace: A Benchmark for Testing Reflection Ability and Controlled Evolution in LLM Agents](https://mlsanity.com/paper/2605.29225/) (2026)
- [FinEvolveBench: A Benchmark for Self-Evolving Agents on Low-Repetition Tasks with Implicit Rewards](https://mlsanity.com/paper/2606.06960/) (2026)
- [Bifrost: Steering Strategic Trajectories to Bridge Contextual Gaps for Self-Improving Agents](https://mlsanity.com/paper/2602.05810/) (2026)
- [Web2BigTable: A Bi-Level Multi-Agent LLM System for Internet-Scale Information Search and Extraction](https://mlsanity.com/paper/2604.27221/) (2026)
- [The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Project](https://mlsanity.com/paper/2603.21354/) (2026)
- [Training Language Agents to Learn from Experience](https://mlsanity.com/paper/2605.20477/) (2026)
- [Success Leaves Detours: Learning Executable Walkthroughs for Long-Horizon Agents](https://mlsanity.com/paper/2609.22120/) (2026)
- [My FAULT: Self-Diagnosis as Credit Assignment in Self-Evolving Agentic Reinforcement Learning](https://mlsanity.com/paper/2610.01161/) (2026)

## Abstract (arXiv)

Despite the rapid advancements in LLM agents, they still face the challenge of generating meaningful reflections due to inadequate error analysis and a reliance on rare successful trajectories, especially in complex tasks. In this work, we propose SAMULE, a new framework for self-learning agents powered by a retrospective language model that is trained based on Multi-Level Reflection Synthesis. It first synthesizes high-quality reflections across three complementary levels: Single-Trajectory Learning (micro-level) for detailed error correction; Intra-Task Learning (meso-level) to build error taxonomies across multiple trials of the same task, and Inter-Task Learning (macro-level) to extract transferable insights based on same typed errors from diverse task failures. Then we fine-tune a language model serving as the retrospective model to generate reflections during inference. We further extend our framework to interactive settings through a foresight-based reflection mechanism, enabling agents to proactively reflect and adapt during user interactions by comparing predicted and actual responses. Extensive experiments on three challenging benchmarks - TravelPlanner, NATURAL PLAN, and Tau-bench - demonstrate that our approach significantly outperforms reflection-based baselines. Our results highlight the critical role of well-designed reflection synthesis and failure-centric learning in building self-improving LLM agents.