mlsanityData to 14 Oct 2026

The Landscape of Agentic Reinforcement Learning for LLMs: A Survey

arXiv 2509.02547 ↗First posted 2 Sep 2025cs.AI cs.CLcited by 205

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

The emergence of agentic reinforcement learning (Agentic RL) marks a paradigm shift from conventional reinforcement learning applied to large language models (LLM RL), reframing LLMs from passive sequence generators into autonomous, decision-making agents embedded in complex, dynamic worlds. This survey formalizes this conceptual shift by contrasting the degenerate single-step Markov Decision Processes (MDPs) of LLM-RL with the temporally extended, partially observable Markov decision processes (POMDPs) that define Agentic RL. Building on this foundation, we propose a comprehensive twofold taxonomy: one organized around core agentic capabilities, including planning, tool use, memory, reasoning, self-improvement, and perception, and the other around their applications across diverse task domains. Central to our thesis is that reinforcement learning serves as the critical mechanism for transforming these capabilities from static, heuristic modules into adaptive, robust agentic behavior. To support and accelerate future research, we consolidate the landscape of open-source environments, benchmarks, and frameworks into a practical compendium. By synthesizing over five hundred recent works, this survey charts the contours of this rapidly evolving field and highlights the opportunities and challenges that will shape the development of scalable, general-purpose AI agents.

Key facts. Cited by 205 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 387 of its 618 references that are in the map. First posted on arXiv 2 Sep 2025.

Builds on

387 of 618 references.

Closest ancestors · 33

Same direction, newest first

  1. 2025Reinforcement Learning for Machine Learning Engineering Agents1 Sep
  2. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use1 Sep
  3. 2025rStar2-Agent: Agentic Reasoning Technical Report28 Aug
  4. 2025AWorld: Orchestrating the Training Recipe for Agentic AI28 Aug
  5. 2025Encouraging Good Processes Without the Need for Good Answers: Reinforcement Learning for LLM Agent Planning27 Aug
  6. 2025Feedback-Driven Tool-Use Improvements in Large Language Models via Automated Build Environments12 Aug
  7. 2025Reasoning through Exploration: A Reinforcement Learning Framework for Robust Function Calling7 Aug
  8. 2025LLM Collaboration With Multi-Agent Reinforcement Learning6 Aug
  9. 2025Agent Lightning: Train ANY AI Agents with Reinforcement Learning5 Aug
  10. 2025Training Long-Context, Multi-Turn Software Engineering Agents with Reinforcement Learning5 Aug
  11. 2025PilotRL: Training Language Model Agents via Global Planning-Guided Progressive Reinforcement Learning1 Aug
  12. 2025AgentFly: Extensible and Scalable Reinforcement Learning for LM Agents20 Jul
21 more
  1. 2025How to Train a Leader: Hierarchical Reasoning in Multi-Agent LLMs11 Jul
  2. 2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning30 Jun
  3. 2025L0: Reinforcement Learning to Become General Agents30 Jun
  4. 2025ARIA: Training Language Agents with Intention-Driven Reward Aggregation31 May
  5. 2025ML-Agent: Reinforcing LLM Agents for Autonomous Machine Learning Engineering29 May
  6. 2025SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attribution27 May
  7. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  8. 2025Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solving12 May
  9. 2025Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learning28 Apr
  10. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning24 Apr
  11. 2025MARFT: Multi-Agent Reinforcement Fine-Tuning21 Apr
  12. 2025Acting Less is Reasoning More! Teaching Model to Act Efficiently21 Apr
  13. 2025ToolRL: Reward is All Tool Learning Needs16 Apr
  14. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMs15 Apr
  15. 2025Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Use7 Apr
  16. 2025ToRL: Scaling Tool-Integrated RL30 Mar
  17. 2025LERO: LLM-driven Evolutionary framework with Hybrid Rewards and Enhanced Observation for Multi-Agent Reinforcement Learning25 Mar
  18. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks19 Mar
  19. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learning12 Mar
  20. 2025Process Reward Models for LLM Agents: Practical Framework and Directions14 Feb
  21. 2024Luban: Building Open-Ended Creative Agents via Autonomous Embodied Verification24 May

Other references · 354

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  8. 2023Let's Verify Step by Stepcited by 3,789
  9. 2021Program Synthesis with Large Language Modelscited by 3,637
  10. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  12. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
38 more
  1. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  2. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  3. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  4. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  5. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  6. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  7. 2024OpenAI o1 System Cardcited by 1,244
  8. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  9. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  10. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  11. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,063
  12. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  13. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  14. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  15. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 872
  16. 2025Group Sequence Policy Optimizationcited by 835
  17. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  18. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 797
  19. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 779
  20. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  21. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  22. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  23. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  24. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 665
  25. 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 661
  26. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 647
  27. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 627
  28. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 610
  29. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  30. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 559
  31. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
  32. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 526
  33. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 513
  34. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 454
  35. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 443
  36. 2025Search-o1: Agentic Search-Enhanced Large Reasoning Modelscited by 433
  37. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Basescited by 429
  38. 2024Understanding the planning of LLM agents: A surveycited by 421

50 most cited of 354 All 354 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 52; 2026: 153.

205 papers.

Sort:

By area:

  1. 2025Memory in the Age of AI Agentscited by 303
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  3. 2025Scaling Long-Horizon LLM Agent via Context-Foldingcited by 148
  4. 2026SoK: Agentic Skills -- Beyond Tool Use in LLM Agentscited by 130
  5. 2025AgentFold: Long-Horizon Web Agents with Proactive Context Managementcited by 103
  6. 2025AgentEvolver: Towards Efficient Self-Evolving Agent Systemcited by 85
  7. 2025Tree Search for LLM Agent Reinforcement Learningcited by 79
  8. 2026Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligencecited by 54
  9. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  10. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Taskscited by 49
  11. 2025Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agentscited by 48
  12. 2025Agentic Entropy-Balanced Policy Optimizationcited by 40
38 more
  1. 2025VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Modelscited by 36
  2. 2025Search Self-play: Pushing the Frontier of Agent Capability without Supervisioncited by 34
  3. 2026Tool-R0: Self-Evolving LLM Agents for Tool-Learning from Zero Datacited by 34
  4. 2026Dynamic Dual-Granularity Skill Bank for Agentic RLcited by 34
  5. 2025UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learningcited by 32
  6. 2026OPID: On-Policy Skill Distillation for Agentic Reinforcement Learningcited by 32
  7. 2026SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learningcited by 32
  8. 2025A Survey of Data Agents: Emerging Paradigm or Overstated Hype?cited by 30
  9. 2025Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoningcited by 29
  10. 2025CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewardscited by 29
  11. 2025Agent-R1: A Unified and Modular Framework for Agentic Reinforcement Learningcited by 29
  12. 2026ARISE: Agent Reasoning with Intrinsic Skill Evolution in Hierarchical Reinforcement Learningcited by 29
  13. 2025MemSearcher: Training LLMs to Reason, Search and Manage Memory via End-to-End Reinforcement Learningcited by 27
  14. 2026Look Inward to Explore Outward: Learning Temperature Policy from LLM Internal States via Hierarchical RLcited by 27
  15. 2025A Survey on Agentic Multimodal Large Language Modelscited by 23
  16. 2026From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Modelscited by 23
  17. 2026AgentOCR: Reimagining Agent History via Optical Self-Compressioncited by 22
  18. 2025Revisiting Entropy in Reinforcement Learning for Large Reasoning Modelscited by 21
  19. 2025Meta-RL Induces Exploration in Language Agentscited by 21
  20. 2026SCRIBE: Structured Mid-Level Supervision for Tool-Using Language Modelscited by 21
  21. 2026Milestone-Guided Policy Learning for Long-Horizon Language Agentscited by 19
  22. 2025LIMI: Less is More for Agencycited by 17
  23. 2025DeepTravel: An End-to-End Agentic Reinforcement Learning Framework for Autonomous Travel Planning Agentscited by 17
  24. 2025AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agentcited by 17
  25. 2026Experience Compression Spectrum: Unifying Memory, Skills, and Rules in LLM Agentscited by 17
  26. 2026Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systemscited by 16
  27. 2025Reinforcement Learning Foundations for Deep Research Systems: A Surveycited by 14
  28. 2025Doctor-R1: Mastering Clinical Inquiry with Experiential Agentic Reinforcement Learningcited by 14
  29. 2026SmartSearch: Process Reward-Guided Query Refinement for Search Agentscited by 14
  30. 2025Environment Scaling for Interactive Agentic Experience Collection: A Surveycited by 13
  31. 2025Agentic Learner with Grow-and-Refine Multimodal Semantic Memorycited by 13
  32. 2026MemoBrain: Executive Memory as an Agentic Brain for Reasoningcited by 13
  33. 2026InfoPO: Information-Driven Policy Optimization for User-Centric Agentscited by 12
  34. 2026RTMC: Step-Level Credit Assignment via Rollout Treescited by 12
  35. 2026NEWTON: Agentic Planning for Physically Grounded Video Generationcited by 12
  36. 2025POLO: Preference-Guided Multi-Turn Reinforcement Learning for Lead Optimizationcited by 11
  37. 2026Spark: Strategic Policy-Aware Exploration via Dynamic Branching for Long-Horizon Agentic Learningcited by 11
  38. 2026Synthetic Sandbox for Training Machine Learning Engineering Agentscited by 11

50 most cited of 205 All 205 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Demystifying Reinforcement Learning in Agentic Reasoning
  2. 2024A Survey On Enhancing Reinforcement Learning in Complex Environments: Insights from Human and LLM Feedback
  3. 2026Hybrid LLM-Augmented Reinforcement Learning Agents for Complex Sequential Decision Tasks
  4. 2025Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AI
  5. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods
  6. 2026RAPO: Expanding Exploration for LLM Agents via Retrieval-Augmented Policy Optimization
  7. 2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applications
  8. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle