mlsanityData to 7 Oct 2026

APEX: Autonomous Policy Exploration for Self-Evolving LLM Agents

arXiv 2605.21240 ↗First posted 20 May 2026cs.LG cs.AIcited by 2

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

also in 1 more
Abstract (arXiv)

LLM agents have shown strong performance across a wide range of complex tasks, including interactive environments that require long-horizon decision making. But these agents cannot learn on the fly at test time. Self-evolving agents address this by accumulating memory and reflection across episodes rather than requiring model-weight updates. However, these agents often suffer from exploration collapse: as memory grows, behavior concentrates around familiar high-reward routines, reducing the chance of discovering better alternatives. To address this problem, we propose Autonomous Policy EXploration (APEX), which builds and maintains an explicit strategy space through a strategy map-a directed acyclic graph of milestones with prerequisite dependency edges. In APEX, Fork Discovery expands the map with evidence-grounded unexplored directions, while Policy Selection balances exploration and exploitation during planning. Evaluated on nine Jericho text-adventure games and WebArena, a realistic web interaction benchmark, APEX outperforms all baselines. Extensive ablations validate each component's contribution and demonstrate robustness across diverse settings, demonstrating APEX's effectiveness for sustained exploration in self-evolving agents.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 20 of its 26 references that are in the map. First posted on arXiv 20 May 2026.

Builds on

20 of 26 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Voyager: An Open-Ended Embodied Agent with Large Language Models25 May
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learning20 Mar

Other references · 18

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  3. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  4. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,002
  5. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  6. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  7. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
  8. 2017A Tutorial on Thompson Samplingcited by 420
  9. 2025Agentic Context Engineering: Evolving Contexts for Self-Improving Language Modelscited by 349
  10. 2024Agent Workflow Memorycited by 342
  11. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  12. 2025ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memorycited by 253
6 more
  1. 2020Never Give Up: Learning Directed Exploration Strategiescited by 210
  2. 2019Interactive Fiction Games: A Colossal Adventurecited by 165
  3. 2025Monte Carlo Tree Search for Comprehensive Exploration in LLM-Based Automatic Heuristic Designcited by 100
  4. 2025EvoTest: Evolutionary Test-Time Learning for Self-Improving Agentic Systemscited by 34
  5. 2018Contingency-Aware Exploration in Reinforcement Learningcited by 26
  6. 2026Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updatescited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026The Last AI Built by Humans: Toward Genuine Recursive Self-Improvementcited by 5
  2. 2026Success Leaves Detours: Learning Executable Walkthroughs for Long-Horizon Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Look Before You Leap: Autonomous Exploration for LLM Agents
  2. 2026Learning to Explore: Scaling Agentic Reasoning via Exploration-Aware Policy Optimization
  3. 2025Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learning
  4. 2026EvoTrainer: Co-Evolving LLM Policies and Training Harnesses for Autonomous Agentic Reinforcement Learning
  5. 2026SEARL: Joint Optimization of Policy and Tool Graph Memory for Self-Evolving Agents
  6. 2026ForkPilot: Self-Evolving Policy for Retrospective Search in Long-Horizon Agents
  7. 2026SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning
  8. 2026Self-Evolving World Models for LLM Agent Planning