# MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memory

arXiv: https://arxiv.org/abs/2601.03192 · HTML: https://mlsanity.com/paper/2601.03192/

## Key facts

- Cited by 129 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 42 of its 76 references that are in the map.
- First posted on arXiv 6 Jan 2026.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 30 of 42)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [A-MEM: Agentic Memory for LLM Agents](https://mlsanity.com/paper/2502.12110/) (2025)
- [Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory](https://mlsanity.com/paper/2504.19413/) (2025)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [Value-Decomposition Networks For Cooperative Multi-Agent Learning](https://mlsanity.com/paper/1706.05296/) (2017)
- [Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2003.08839/) (2020)
- [Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review](https://mlsanity.com/paper/1805.00909/) (2018)
- [A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts](https://mlsanity.com/paper/2303.15361/) (2023)
- [The Power of Noise: Redefining Retrieval for RAG Systems](https://mlsanity.com/paper/2401.14887/) (2024)
- [Neural Episodic Control](https://mlsanity.com/paper/1703.01988/) (2017)
- [Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory](https://mlsanity.com/paper/2511.20857/) (2025)
- [Memento: Fine-tuning LLM Agents without Fine-tuning LLMs](https://mlsanity.com/paper/2508.16153/) (2025)

## Built on by (most cited first, 30 of 129)

- [SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning](https://mlsanity.com/paper/2602.08234/) (2026)
- [SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalization](https://mlsanity.com/paper/2604.02268/) (2026)
- [SkillClaw: Let Skills Evolve Collectively with Agentic Evolver](https://mlsanity.com/paper/2604.08377/) (2026)
- [Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering](https://mlsanity.com/paper/2604.08224/) (2026)
- [A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agents](https://mlsanity.com/paper/2601.12538/) (2026)
- [SkillOS: Learning Skill Curation for Self-Evolving Agents](https://mlsanity.com/paper/2605.06614/) (2026)
- [Skill-Pro: Learning Reusable Skills from Experience via Non-Parametric PPO for LLM Agents](https://mlsanity.com/paper/2602.01869/) (2026)
- [MetaClaw: Just Talk -- An Agent That Meta-Learns and Evolves in the Wild](https://mlsanity.com/paper/2603.17187/) (2026)
- [Dynamic Dual-Granularity Skill Bank for Agentic RL](https://mlsanity.com/paper/2603.28716/) (2026)
- [Self-Improvements in Modern Agentic Systems: A Survey](https://mlsanity.com/paper/2607.13104/) (2026)
- [Mem-T: Densifying Rewards for Long-Horizon Memory Agents](https://mlsanity.com/paper/2601.23014/) (2026)
- [Co-Evolving LLM Decision and Skill Bank Agents for Long-Horizon Tasks](https://mlsanity.com/paper/2604.20987/) (2026)
- [SkillGraph: Skill-Augmented Reinforcement Learning for Agents via Evolving Skill Graphs](https://mlsanity.com/paper/2605.12039/) (2026)
- [EvolveMem:Self-Evolving Memory Architecture via AutoResearch for LLM Agents](https://mlsanity.com/paper/2605.13941/) (2026)
- [ReSkill: Reconciling Skill Creation with Policy Optimization in Agentic RL](https://mlsanity.com/paper/2606.01619/) (2026)
- [MemMA: Coordinating the Memory Cycle through Multi-Agent Reasoning and In-Situ Self-Evolution](https://mlsanity.com/paper/2603.18718/) (2026)
- [CODESKILL: Learning Self-Evolving Skills for Coding Agents](https://mlsanity.com/paper/2605.25430/) (2026)
- [SPADE: Self-Play in Adaptive Synthetic Executable Environments](https://mlsanity.com/paper/2608.19197/) (2026)
- [Belief Memory: Agent Memory Under Partial Observability](https://mlsanity.com/paper/2605.05583/) (2026)
- [TAME: A Trustworthy Test-Time Evolution of Agent Memory with Systematic Benchmarking](https://mlsanity.com/paper/2602.03224/) (2026)
- [Skills on the Fly: Test-Time Adaptive Skill Synthesis for LLM Agents](https://mlsanity.com/paper/2605.16986/) (2026)
- [Auto-Dreamer: Learning Offline Memory Consolidation for Language Agents](https://mlsanity.com/paper/2605.20616/) (2026)
- [Skill0.5: Joint Skill Internalization and Utilization for Out-of-Distribution Generalization in Agentic Reinforcement Learning](https://mlsanity.com/paper/2605.28424/) (2026)
- [PAST-Bench: Benchmarking the Foundations of Recursive Self-Improvement in Personal Agents](https://mlsanity.com/paper/2608.04003/) (2026)
- [UMEM: Unified Memory Extraction and Management Framework for Generalizable Memory](https://mlsanity.com/paper/2602.10652/) (2026)
- [SkillMaster: Toward Autonomous Skill Mastery in LLM Agents](https://mlsanity.com/paper/2605.08693/) (2026)
- [Safety in Self-Evolving LLM Agent Systems: Threats, Amplification, and Case Studies](https://mlsanity.com/paper/2606.23075/) (2026)
- [MemReranker: Reasoning-Aware Reranking for Agent Memory Retrieval](https://mlsanity.com/paper/2605.06132/) (2026)
- [VideoWeaver: Evaluating and Evolving Skills for Agentic Long Video Generation](https://mlsanity.com/paper/2606.08091/) (2026)
- [Co-Evolving Skill Generation and Policy Optimization](https://mlsanity.com/paper/2606.08755/) (2026)

## Abstract (arXiv)

The hallmark of human intelligence is the self-evolving ability to master new skills by learning from past experiences. However, current AI agents struggle to emulate this self-evolution: fine-tuning is computationally expensive and prone to catastrophic forgetting, while existing memory-based methods rely on passive semantic matching that often retrieves noise. To address these challenges, we propose MemRL, a non-parametric approach that evolves via reinforcement learning on episodic memory. By decoupling stable reasoning from plastic memory, MemRL employs a Two-Phase Retrieval mechanism to filter noise and identify high-utility strategies through environmental feedback. Extensive experiments on HLE, BigCodeBench, ALFWorld, and Lifelong Agent Bench demonstrate that MemRL significantly outperforms state-of-the-art baselines, confirming that MemRL effectively reconciles the stability-plasticity dilemma, enabling continuous runtime improvement without weight updates. Code is available at https://github.com/MemTensor/MemRL.