mlsanityData to 7 Oct 2026

Reinforcement Learning for Long-Horizon Interactive LLM Agents

arXiv 2502.01600 ↗First posted 3 Feb 2025cs.LG cs.AIcited by 103

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Interactive digital agents (IDAs) leverage APIs of stateful digital environments to perform tasks in response to user requests. While IDAs powered by instruction-tuned large language models (LLMs) can react to feedback from interface invocations in multi-step exchanges, they have not been trained in their respective digital environments. Prior methods accomplish less than half of tasks in sophisticated benchmarks such as AppWorld. We present a reinforcement learning (RL) approach that trains IDAs directly in their target environments. We formalize this training as a partially observable Markov decision process and derive LOOP, a data- and memory-efficient variant of proximal policy optimization. LOOP uses no value network and maintains exactly one copy of the underlying LLM in memory, making its implementation straightforward and as memory-efficient as fine-tuning a single LLM. A 32-billion-parameter agent trained with LOOP in the AppWorld environment outperforms the much larger OpenAI o1 agent by 9 percentage points (15% relative). To our knowledge, this is the first reported application of RL to IDAs that interact with a stateful, multi-domain, multi-app environment via direct API calls. Our analysis sheds light on the effectiveness of RL in this area, showing that the agent learns to consult the API documentation, avoid unwarranted assumptions, minimize confabulation, and recover from setbacks.

Key facts. Cited by 103 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 16 of its 49 references that are in the map. First posted on arXiv 3 Feb 2025.

Builds on

16 of 49 references.

References · 16

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  5. 2024Qwen2.5 Technical Reportcited by 4,713
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  7. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  8. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  9. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  10. 2023FireAct: Toward Language Agent Fine-tuningcited by 247
  11. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 210
  12. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 201
4 more
  1. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
  2. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 158
  3. 2024AgentInstruct: Toward Generative Teaching with Agentic Flowscited by 88
  4. 2024Cut Your Losses in Large-Vocabulary Language Modelscited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 30; 2026: 73.

103 papers.

Sort:

By area:

  1. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  2. 2025Reinforcement Learning for Self-Improving Agent with Skill Librarycited by 91
  3. 2026OpenClaw-RL: Train Any Agent Simply by Talkingcited by 83
  4. 2025Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignmentcited by 77
  5. 2026Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agentscited by 75
  6. 2025WebSailor-V2: Bridging the Chasm to Proprietary Agents via Synthetic Data and Scalable Reinforcement Learningcited by 59
  7. 2025AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Frameworkcited by 58
  8. 2025Tongyi DeepResearch Technical Reportcited by 53
  9. 2025REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websitescited by 43
  10. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Taskscited by 37
  11. 2026Vision-DeepResearch: Incentivizing DeepResearch Capability in Multimodal Large Language Modelscited by 33
  12. 2025A Simple and Effective Reinforcement Learning Method for Text-to-Image Diffusion Fine-tuningcited by 30
38 more
  1. 2025Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Gamescited by 30
  2. 2026Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learningcited by 28
  3. 2026The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Breakcited by 25
  4. 2025SALT: Step-level Advantage Assignment for Long-horizon Agents via Trajectory Graphcited by 22
  5. 2026On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agentscited by 21
  6. 2026SOD: Step-wise On-policy Distillation for Small Language Model Agentscited by 17
  7. 2025RobustVLA: Robustness-Aware Reinforcement Post-Training for Vision-Language-Action Modelscited by 16
  8. 2026RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedbackcited by 16
  9. 2025Dyna-Think: Synergizing Reasoning, Acting, and World Model Simulation in AI Agentscited by 14
  10. 2026HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agentscited by 14
  11. 2026A Survey on LLM-based Conversational User Simulationcited by 14
  12. 2026AtomMem : Learnable Dynamic Agentic Memory with Atomic Memory Operationcited by 13
  13. 2025World Modelling Improves Language Model Agentscited by 12
  14. 2025GTM: Simulating the World of Tools for AI Agentscited by 12
  15. 2025Kinetics: Rethinking Test-Time Scaling Lawscited by 11
  16. 2025A Survey of Scaling in Large Language Model Reasoningcited by 10
  17. 2025Benchmarking the Robustness of Agentic Systems to Adversarially-Induced Harmscited by 10
  18. 2026Entropy-Preserving Reinforcement Learningcited by 10
  19. 2026AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluationcited by 10
  20. 2026EvoTrainer: Co-Evolving LLM Policies and Training Harnesses for Autonomous Agentic Reinforcement Learningcited by 9
  21. 2025Behavior Injection: Preparing Language Models for Reinforcement Learningcited by 8
  22. 2026Budget-Constrained Agentic Large Language Models: Intention-Based Planning for Costly Tool Usecited by 8
  23. 2025A Course Correction in Steerability Evaluation: Revealing Miscalibration and Side Effects in LLMscited by 7
  24. 2025Planner-R1: Reward Shaping Enables Efficient Agentic RL with Smaller LLMscited by 7
  25. 2026Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learningcited by 7
  26. 2025Don't Adapt Small Language Models for Tools; Adapt Tool Schemas to the Modelscited by 6
  27. 2025FoldAct: Efficient and Stable Context Folding for Long-Horizon Search Agentscited by 6
  28. 2026GAGPO: Generalized Advantage Grouped Policy Optimizationcited by 6
  29. 2026SkillFlow: Flow-Driven Recursive Skill Evolution for Agentic Orchestrationcited by 6
  30. 2025From Benchmarks to Business Impact: Deploying IBM Generalist Agent in Enterprise Productioncited by 5
  31. 2026CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVRcited by 5
  32. 2026On Training Large Language Models for Long-Horizon Tasks: An Empirical Study of Horizon Lengthcited by 5
  33. 2026Resolving Action Bottleneck: Agentic Reinforcement Learning Informed by Token-Level Energycited by 5
  34. 2026SIRI: Self-Internalizing Reinforcement Learning with Intrinsic Skills for LLM Agent Trainingcited by 5
  35. 2025Leveraging In-Context Learning for Language Model Agentscited by 4
  36. 2025Enhancing Vision-Language Model Training with Reinforcement Learning in Synthetic Worlds for Real-World Successcited by 4
  37. 2025Code-enabled language models can outperform reasoning models on diverse taskscited by 4
  38. 2025Tailored Primitive Initialization is the Secret Key to Reinforcement Learningcited by 4

50 most cited of 103 All 103 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Claw-R1: A Step-Level Data Middleware System for Agentic Reinforcement Learning
  2. 2023Enabling Intelligent Interactions between an Agent and an LLM: A Reinforcement Learning Approach
  3. 2025AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning
  4. 2025Agent-R1: A Unified and Modular Framework for Agentic Reinforcement Learning
  5. 2025A Practitioner's Guide to Multi-turn Agentic Reinforcement Learning
  6. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Survey
  7. 2026Rethinking Agentic Reinforcement Learning In Large Language Models
  8. 2026A Subgoal-driven Framework for Improving Long-Horizon LLM Agents