mlsanityData to 7 Oct 2026

Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimization

arXiv 2602.23008 ↗First posted 26 Feb 2026cs.LG cs.AIcited by 17

On the map: LLM agents and tool use → Agent memory systems

Abstract (arXiv)

Exploration remains the key bottleneck for large language model agents trained with reinforcement learning. While prior methods exploit pretrained knowledge, they fail in environments requiring the discovery of novel states. We propose Exploratory Memory-Augmented On- and Off-Policy Optimization (EMPO2), a hybrid RL framework that leverages memory for exploration and combines on- and off-policy updates to make LLMs perform well with memory while also ensuring robustness without it. On ScienceWorld and WebShop, EMPO2 achieves 128.6% and 11.3% improvements over GRPO, respectively. Moreover, in out-of-distribution tests, EMPO2 demonstrates superior adaptability to new tasks, requiring only a few trials with memory and no parameter updates. These results highlight EMPO2 as a promising framework for building more exploratory and generalizable LLM-based agents.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Agent memory systems, one of 16 directions in LLM agents and tool use. Builds on 39 of its 42 references that are in the map. First posted on arXiv 26 Feb 2026.

Builds on

39 of 42 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Large Language Models Are Semi-Parametric Reinforcement Learning Agents9 Jun

Other references · 38

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  7. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  10. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  11. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  12. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
26 more
  1. 2018Exploration by Random Network Distillationcited by 966
  2. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  3. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  4. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 377
  5. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
  6. 2018TextWorld: A Learning Environment for Text-based Gamescited by 304
  7. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 298
  8. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 207
  9. 2024Trial and Error: Exploration-Based Trajectory Optimization for LLM Agentscited by 194
  10. 2023SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Taskscited by 166
  11. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learningcited by 166
  12. 2025Agentic Reinforced Policy Optimizationcited by 166
  13. 2022Learning by Distilling Contextcited by 128
  14. 2024WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environmentcited by 99
  15. 2025Agent Lightning: Train ANY AI Agents with Reinforcement Learningcited by 94
  16. 2024Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinementcited by 86
  17. 2024Agent Planning with World Knowledge Modelcited by 66
  18. 2025RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agentscited by 56
  19. 2025Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMscited by 43
  20. 2024Intelligent Go-Explore: Standing on the Shoulders of Giant Foundation Modelscited by 32
  21. 2025ReflAct: World-Grounded Decision Making in LLM Agents via Goal-State Reflectioncited by 30
  22. 2025Training a Generally Curious Agentcited by 29
  23. 2024WALL-E: World Alignment by Rule Learning Improves World Model-based LLM Agentscited by 24
  24. 2024Better than Your Teacher: LLM Agents that learn from Privileged AI Feedbackcited by 21
  25. 2025Retrospex: Language Agent Meets Offline Reinforcement Learning Criticcited by 12
  26. 2025Agentic Knowledgeable Self-awarenesscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 17.

17 papers.

Sort:

By area:

  1. 2026Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?cited by 87
  2. 2026Dynamic Dual-Granularity Skill Bank for Agentic RLcited by 33
  3. 2026Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learningcited by 28
  4. 2026From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanismscited by 19
  5. 2026RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedbackcited by 16
  6. 2026Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVRcited by 13
  7. 2026Understanding Reasoning in LLMs through Strategic Information Allocation under Uncertaintycited by 8
  8. 2026From History to State: Constant-Context Skill Learning for LLM Agentscited by 5
  9. 2026AdaMEM: Test-Time Adaptive Memory for Language Agentscited by 4
  10. 2026LeanMem: Simple and Efficient Long-Term Memory for LLM Agentscited by 4
  11. 2026RODS: Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agentscited by 2
  12. 2026EDGE: Experience-Distillation for Guided Exploration in Agentic Reinforcement Learningcited by 2
5 more
  1. 2026MAP: A Map-then-Act Paradigm for Long-Horizon Interactive Agent Reasoningcited by 1
  2. 2026AIPO: Learning to Reason from Active Interactioncited by 0
  3. 2026Personalizing Embodied Multimodal Large Language Model Agents over Long-term User Interactionscited by 0
  4. 2026Success Leaves Detours: Learning Executable Walkthroughs for Long-Horizon Agentscited by 0
  5. 2026TTSE: A Two-Track Online Self-Evolution Frameworkcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026RAPO: Expanding Exploration for LLM Agents via Retrieval-Augmented Policy Optimization
  2. 2026From Player to Master: Enhancing Test-Time Learning of LLM Agents via Reinforcement Learning over Memory
  3. 2026Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontiers
  4. 2025EEPO: Exploration-Enhanced Policy Optimization via Sample-Then-Forget
  5. 2026MemFactory: Unified Inference & Training Framework for Agent Memory
  6. 2025Toward Efficient Exploration by Large Language Model Agents
  7. 2024Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement
  8. 2025Memento: Fine-tuning LLM Agents without Fine-tuning LLMs