mlsanityData to 7 Oct 2026

RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning

arXiv 2504.20073 ↗First posted 24 Apr 2025cs.LG cs.AI cs.CLcited by 357

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Training large language models (LLMs) as interactive agents presents unique challenges including long-horizon decision making and interacting with stochastic environment feedback. While reinforcement learning (RL) has enabled progress in static tasks, multi-turn agent RL training remains underexplored. We propose StarPO (State-Thinking-Actions-Reward Policy Optimization), a general framework for trajectory-level agent RL, and introduce RAGEN, a modular system for training and evaluating LLM agents. Our study on four stylized environments reveals three core findings. First, our agent RL training shows a recurring mode of Echo Trap where reward variance cliffs and gradient spikes; we address this with StarPO-S, a stabilized variant with trajectory filtering, critic incorporation, and gradient stabilization. Second, we find the shaping of RL rollouts would benefit from diverse initial states, medium interaction granularity and more frequent sampling. Third, we show that without fine-grained, reasoning-aware reward signals, agent reasoning hardly emerge through multi-turn RL and they may show shallow strategies or hallucinated thoughts. Code and environments are available at https://github.com/RAGEN-AI/RAGEN.

Key facts. Cited by 357 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 47 of its 66 references that are in the map. First posted on arXiv 24 Apr 2025.

Builds on

47 of 66 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Reflexion: Language Agents with Verbal Reinforcement Learning20 Mar

Other references · 46

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  8. 2023Let's Verify Step by Stepcited by 3,598
  9. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  10. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  11. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  12. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
34 more
  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  2. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  3. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  4. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  5. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  6. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  7. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  8. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  9. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 580
  10. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 505
  11. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  12. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 450
  13. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  14. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 402
  15. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 325
  16. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  17. 2023Think before you speak: Training Language Models With Pause Tokenscited by 254
  18. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 238
  19. 2024Let's Think Dot by Dot: Hidden Computation in Transformer Language Modelscited by 186
  20. 2023ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Modelscited by 174
  21. 2023SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Taskscited by 166
  22. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 158
  23. 2024Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?cited by 153
  24. 2024Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputscited by 148
  25. 2023OpenAgents: An Open Platform for Language Agents in the Wildcited by 136
  26. 2023ToolChain*: Efficient Action Space Navigation in Large Language Models with A* Searchcited by 103
  27. 2023AgentSims: An Open-Source Sandbox for Large Language Model Evaluationcited by 86
  28. 2023Reason for Future, Act for Now: A Principled Framework for Autonomous LLM Agents with Provable Sample Efficiencycited by 46
  29. 2024Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHFcited by 29
  30. 2024Thinking Tokens for Language Modelingcited by 23
  31. 2025EnigmaEval: A Benchmark of Long Multimodal Reasoning Challengescited by 19
  32. 2024Enhancing LLM Reasoning with Multi-Path Collaborative Reactive and Reflection agentscited by 15
  33. 2023Advances in Embodied Navigation Using Large Language Models: A Surveycited by 14
  34. 2024Teaching Embodied Reinforcement Learning Agents: Informativeness and Diversity of Language Usecited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 119; 2026: 238.

357 papers.

Sort:

By area:

  1. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  2. 2025WebThinker: Empowering Large Reasoning Models with Deep Research Capabilitycited by 320
  3. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  4. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  5. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  6. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learningcited by 166
  7. 2025Agentic Reinforced Policy Optimizationcited by 166
  8. 2025WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learningcited by 162
  9. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  10. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoningcited by 145
  11. 2026MemSkill: Learning and Evolving Memory Skills for Self-Evolving Agentscited by 129
  12. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 121
38 more
  1. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Librarycited by 94
  2. 2025Agent Lightning: Train ANY AI Agents with Reinforcement Learningcited by 94
  3. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Usecited by 92
  4. 2025Reinforcement Learning for Self-Improving Agent with Skill Librarycited by 91
  5. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  6. 2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learningcited by 89
  7. 2025Grounded Reinforcement Learning for Visual Reasoningcited by 80
  8. 2025Agent Learning via Early Experiencecited by 79
  9. 2025Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignmentcited by 77
  10. 2025ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replaycited by 75
  11. 2026Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agentscited by 75
  12. 2025Tree Search for LLM Agent Reinforcement Learningcited by 73
  13. 2025SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attributioncited by 68
  14. 2025VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learningcited by 68
  15. 2025In-the-Flow Agentic System Optimization for Effective Planning and Tool Usecited by 67
  16. 2025VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agentscited by 60
  17. 2025AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Frameworkcited by 58
  18. 2025lmgame-Bench: How Good are LLMs at Playing Games?cited by 54
  19. 2025Simulating Environments with Reasoning Models for Agent Trainingcited by 52
  20. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  21. 2025Agentic Reinforcement Learning with Implicit Step Rewardscited by 45
  22. 2025An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agentscited by 43
  23. 2025s3: You Don't Need That Much Data to Train a Search Agent via RLcited by 41
  24. 2025Agentic Entropy-Balanced Policy Optimizationcited by 39
  25. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Taskscited by 37
  26. 2025VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Modelscited by 35
  27. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 34
  28. 2025UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learningcited by 32
  29. 2025Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Gamescited by 30
  30. 2025MUA-RL: Multi-turn User-interacting Agent Reinforcement Learning for agentic tool usecited by 30
  31. 2025PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifiercited by 28
  32. 2025Why Do LLM Agents Fail in Exploring New Environments? A World-Modeling Perspectivecited by 28
  33. 2025Agent-R1: A Unified and Modular Framework for Agentic Reinforcement Learningcited by 28
  34. 2025Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMscited by 28
  35. 2026Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learningcited by 28
  36. 2025UserRL: Training Interactive User-Centric Agent via Reinforcement Learningcited by 27
  37. 2025ML-Agent: Reinforcing LLM Agents for Autonomous Machine Learning Engineeringcited by 26
  38. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26

50 most cited of 357 All 357 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking Agentic Reinforcement Learning In Large Language Models
  2. 2025Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agents
  3. 2025AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning
  4. 2025Multi-Agent Evolve: LLM Self-Improve through Co-evolution
  5. 2026Demystifying Reinforcement Learning for Long-Horizon Tool-Using Agents: A Comprehensive Recipe
  6. 2026PlanPO: Group Planning-Aware Policy Optimization for Multi-Turn Agentic LLMs
  7. 2026RAPO: Expanding Exploration for LLM Agents via Retrieval-Augmented Policy Optimization
  8. 2026SRPO: Setwise Relative Policy Optimization for Multi-Agent LLMs