mlsanityData to 14 Oct 2026

ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory

arXiv 2509.25140 ↗First posted 29 Sep 2025cs.AI cs.CLcited by 284

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 2 more
Abstract (arXiv)

With the growing adoption of large language model agents in persistent real-world roles, they naturally encounter continuous streams of tasks. A key limitation, however, is their failure to learn from the accumulated interaction history, forcing them to discard valuable insights and repeat past errors. We propose ReasoningBank, a novel memory framework that distills generalizable reasoning strategies from an agent's self-judged successful and failed experiences. At test time, an agent retrieves relevant memories from ReasoningBank to inform its interaction and then integrates new learnings back, enabling it to become more capable over time. Building on this powerful experience learner, we further introduce memory-aware test-time scaling (MaTTS), which accelerates and diversifies this learning process by scaling up the agent's interaction experience. By allocating more compute to each task, the agent generates abundant, diverse experiences that provide rich contrastive signals for synthesizing higher-quality memory. The better memory in turn guides more effective scaling, establishing a powerful synergy between memory and test-time scaling. Across web browsing and software engineering benchmarks, ReasoningBank consistently outperforms existing memory mechanisms that store raw trajectories or only successful task routines, improving both effectiveness and efficiency; MaTTS further amplifies these gains. These findings establish memory-driven experience scaling as a new scaling dimension, enabling agents to self-evolve with emergent behaviors naturally arise. Our code can be found at https://github.com/google-research/reasoning-bank.

Key facts. Cited by 284 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 67 of its 81 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

67 of 81 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent3 Jul
  2. 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents18 Jun
  3. 2025Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction9 Jun
  4. 2025Two Heads are Better Than One: Test-time Scaling of Multi-agent Collaborative Reasoning14 Apr
  5. 2025Z1: Efficient Test-time Scaling with Code1 Apr
  6. 2025S*: Test Time Scaling for Code Generation20 Feb
  7. 2024ExACT: Teaching AI Agents to Explore with Reflective-MCTS and Exploratory Learning2 Oct
  8. 2024HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Model18 Aug

Other references · 59

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  4. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  5. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  6. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  7. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  8. 2024A Survey on LLM-as-a-Judgecited by 1,423
  9. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  10. 2025s1: Simple test-time scalingcited by 1,341
  11. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  12. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
38 more
  1. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,063
  2. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  3. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 876
  4. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  5. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 779
  6. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
  7. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 634
  8. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 556
  9. 2024Agent Workflow Memorycited by 383
  10. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353
  11. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 254
  12. 2023Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Controlcited by 187
  13. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 170
  14. 2025Dynamic Cheatsheet: Test-Time Learning with Adaptive Memorycited by 163
  15. 2024The BrowserGym Ecosystem for Web Agent Researchcited by 161
  16. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 137
  17. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  18. 2025Inducing Programmatic Skills for Agentic Taskscited by 111
  19. 2025In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agentscited by 106
  20. 2025Memp: Exploring Agent Procedural Memorycited by 103
  21. 2025Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in Realistic Environmentscited by 95
  22. 2025Gemini Embedding: Generalizable Embeddings from Geminicited by 94
  23. 2025Agent KB: Leveraging Cross-Domain Experience for Agentic Problem Solvingcited by 94
  24. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 89
  25. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 84
  26. 2024RAP: Retrieval-Augmented Planning with Contextual Memory for Multimodal LLM Agentscited by 80
  27. 2025ReasonIR: Training Retrievers for Reasoning Taskscited by 77
  28. 2024AutoGuide: Automated Generation and Selection of Context-Aware Guidelines for Large Language Model Agentscited by 74
  29. 2025MemOS: An Operating System for Memory-Augmented Generation (MAG) in Large Language Modelscited by 73
  30. 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 69
  31. 2025SWE-Exp: Experience-Driven Software Issue Resolutioncited by 65
  32. 2025Scaling Test-time Compute for LLM Agentscited by 63
  33. 2024Human-inspired Episodic Memory for Infinite Context LLMscited by 62
  34. 2025M+: Extending MemoryLLM with Scalable Long-Term Memorycited by 54
  35. 2024Self-evolving Agents with reflective and memory-augmented abilitiescited by 52
  36. 2025Contextual Experience Replay for Self-Improvement of Language Agentscited by 48
  37. 2024StreamBench: Towards Benchmarking Continuous Improvement of Language Agentscited by 47
  38. 2024In-Context Principle Learning from Mistakescited by 44

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 271.

284 papers.

Sort:

By area:

  1. 2025Memory in the Age of AI Agentscited by 303
  2. 2026SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learningcited by 278
  3. 2026Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skillscited by 187
  4. 2026SimpleMem: Efficient Lifelong Memory for LLM Agentscited by 149
  5. 2026MemSkill: Learning and Evolving Memory Skills for Self-Evolving Agentscited by 142
  6. 2025MemEvolve: Meta-Evolution of Agent Memory Systemscited by 95
  7. 2026MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Taskscited by 84
  8. 2026SkillClaw: Let Skills Evolve Collectively with Agentic Evolvercited by 81
  9. 2026SkillX: Automatically Constructing Skill Knowledge Bases for Agentscited by 76
  10. 2026XSkill: Continual Learning from Experience and Skills in Multimodal Agentscited by 75
  11. 2025FLEX: Continuous Agent Evolution via Forward Learning from Experiencecited by 52
  12. 2026CORAL: Towards Autonomous Multi-Agent Evolution for Open-Ended Discoverycited by 52
38 more
  1. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  2. 2026SkillOS: Learning Skill Curation for Self-Evolving Agentscited by 49
  3. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 44
  4. 2026MetaClaw: Just Talk -- An Agent That Meta-Learns and Evolves in the Wildcited by 37
  5. 2026Dynamic Dual-Granularity Skill Bank for Agentic RLcited by 34
  6. 2026Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agentscited by 34
  7. 2025AccelOpt: A Self-Improving LLM Agentic System for AI Accelerator Kernel Optimizationcited by 23
  8. 2026LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleaguescited by 23
  9. 2025Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skillscited by 22
  10. 2026Self-Improvements in Modern Agentic Systems: A Surveycited by 19
  11. 2025SCOPE: Prompt Evolution for Enhancing Agent Effectivenesscited by 18
  12. 2026Learning to Continually Learn via Meta-learning Agentic Memory Designscited by 18
  13. 2026A Comprehensive Survey on Agent Skills: Taxonomy, Techniques, and Applicationscited by 18
  14. 2026SkillsVote: Lifecycle Governance of Agent Skills from Collection, Recommendation to Evolutioncited by 18
  15. 2026Dream-RSI: Recursive Self-Improvement through Evolving Worldscited by 18
  16. 2026EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspectivecited by 17
  17. 2025WebCoach: Self-Evolving Web Agents with Cross-Session Memory Guidancecited by 16
  18. 2026Toward Efficient Agents: Memory, Tool learning, and Planningcited by 16
  19. 2026Position: Agentic Evolution is the Path to Evolving LLMscited by 16
  20. 2025DocLens : A Tool-Augmented Multi-Agent Framework for Long Visual Document Understandingcited by 15
  21. 2026SkillGraph: Skill-Augmented Reinforcement Learning for Agents via Evolving Skill Graphscited by 15
  22. 2026AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Tracescited by 15
  23. 2025AlphaOPT: Formulating Optimization Programs with Self-Improving LLM Experience Librarycited by 14
  24. 2026Explore with Long-term Memory: A Benchmark and Multimodal LLM-based Reinforcement Learning Framework for Embodied Explorationcited by 13
  25. 2026EnvHarness: Awakening Static Worlds for Agent Learningcited by 13
  26. 2025AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agentscited by 12
  27. 2026Memex(RL): Scaling Long-Horizon LLM Agents via Indexed Experience Memorycited by 12
  28. 2026Memory in the LLM Era: Modular Architectures and Strategies in a Unified Frameworkcited by 12
  29. 2026CODESKILL: Learning Self-Evolving Skills for Coding Agentscited by 12
  30. 2026ClawArena: Benchmarking AI Agents in Evolving Information Environmentscited by 11
  31. 2026M^star: Every Task Deserves Its Own Memory Harnesscited by 11
  32. 2026TAME: A Trustworthy Test-Time Evolution of Agent Memory with Systematic Benchmarkingcited by 10
  33. 2026Auto-Dreamer: Learning Offline Memory Consolidation for Language Agentscited by 10
  34. 2026PRO-LONG: Programmatic Memory Enables Long-Horizon Reasoningcited by 10
  35. 2026Do Not Waste Your Rollouts: Recycling Search Experience for Efficient Test-Time Scalingcited by 9
  36. 2026Combee: Scaling Prompt Learning for Self-Improving Language Model Agentscited by 9
  37. 2026Evolving-RL: End-to-End Optimization of Experience-Driven Self-Evolving Capability within Agentscited by 9
  38. 2026Recursive Experiential-Working Memory Evolution for Long-Horizon Agent Harnessescited by 9

50 most cited of 284 All 284 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026TMAS: Scaling Test-Time Compute via Multi-Agent Synergy
  2. 2026Structurally Aligned Subtask-Level Memory for Software Engineering Agents
  3. 2026SAM: State-Adaptive Memory for Long-Horizon Reasoning Agent
  4. 2026ASTER: Agentic Scaling with Tool-integrated Extended Reasoning
  5. 2025MemGen: Weaving Generative Latent Memory for Self-Evolving Agents
  6. 2026Agent Memory Distillation: Empowering Small LLM Agents with Hierarchical Teacher Memory
  7. 2026ContextWeaver: Selective and Dependency-Structured Memory Construction for LLM Agents
  8. 2025Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory