mlsanityData to 7 Oct 2026

ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL

arXiv 2402.19446 ↗First posted 29 Feb 2024cs.LG cs.AI cs.CLcited by 212

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

A broad use case of large language models (LLMs) is in goal-directed decision-making tasks (or "agent" tasks), where an LLM needs to not just generate completions for a given prompt, but rather make intelligent decisions over a multi-turn interaction to accomplish a task (e.g., when interacting with the web, using tools, or providing customer support). Reinforcement learning (RL) provides a general paradigm to address such agent tasks, but current RL methods for LLMs largely focus on optimizing single-turn rewards. By construction, most single-turn RL methods cannot endow LLMs with the ability to intelligently seek information over multiple turns, perform credit assignment, or reason about their past actions -- all of which are critical in agent tasks. This raises the question: how can we design effective and efficient multi-turn RL algorithms for LLMs? In this paper, we develop a framework for building multi-turn RL algorithms for fine-tuning LLMs, that preserves the flexibility of existing single-turn RL methods for LLMs (e.g., proximal policy optimization), while accommodating multiple turns, long horizons, and delayed rewards effectively. To do this, our framework adopts a hierarchical RL approach and runs two RL algorithms in parallel: a high-level off-policy value-based RL algorithm to aggregate reward over utterances, and a low-level RL algorithm that utilizes this high-level value function to train a token policy within each utterance or turn. Our hierarchical framework, Actor-Critic Framework with a Hierarchical Structure (ArCHer), can also give rise to other RL methods. Empirically, we find that ArCHer significantly improves efficiency and performance on agent tasks, attaining a sample efficiency of about 100x over existing methods, while also improving with larger model capacity (upto the 7 billion scale that we tested on).

Key facts. Cited by 212 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 49 of its 65 references that are in the map. First posted on arXiv 29 Feb 2024.

Builds on

49 of 65 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Voyager: An Open-Ended Embodied Agent with Large Language Models25 May
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learning20 Mar

Other references · 47

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  10. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  11. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  12. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
35 more
  1. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  2. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  3. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  4. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  5. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  6. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  7. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
  8. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  9. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  10. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  11. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  12. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  13. 2016The Option-Critic Architecturecited by 433
  14. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 414
  15. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  16. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328
  17. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  18. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  19. 2023FireAct: Toward Language Agent Fine-tuningcited by 247
  20. 2023Pretraining Language Models with Human Preferencescited by 245
  21. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  22. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  23. 2019Interactive Fiction Games: A Colossal Adventurecited by 165
  24. 2019Planning with Goal-Conditioned Policiescited by 144
  25. 2018NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating Systemcited by 97
  26. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 92
  27. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximationcited by 69
  28. 2018Learning to Extract Coherent Summary via Deep Reinforcement Learningcited by 63
  29. 2020ConvLab-2: An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue Systemscited by 52
  30. 2022CICERO: A Dataset for Contextualized Commonsense Inference in Dialoguescited by 38
  31. 2023Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversationscited by 38
  32. 2022CHAI: A CHatbot AI for Task-Oriented Dialogue with Offline Reinforcement Learningcited by 33
  33. 2019ConvLab: Multi-Domain End-to-End Dialog System Platformcited by 29
  34. 2022When is Realizability Sufficient for Off-Policy Reinforcement Learning?cited by 17
  35. 2023Offline Data Enhanced On-Policy Policy Gradient with Provable Guaranteescited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 25; 2025: 87; 2026: 100.

212 papers.

Sort:

By area:

  1. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  2. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 501
  3. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  4. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  5. 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 300
  6. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
  7. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
  8. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 210
  9. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 201
  10. 2025Agentic Reinforced Policy Optimizationcited by 166
  11. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 162
  12. 2025WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learningcited by 162
38 more
  1. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  2. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 146
  3. 2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environmentscited by 120
  4. 2025CollabLLM: From Passive Responders to Active Collaboratorscited by 117
  5. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 108
  6. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 108
  7. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Librarycited by 94
  8. 2025Agent Lightning: Train ANY AI Agents with Reinforcement Learningcited by 94
  9. 2026OpenClaw-RL: Train Any Agent Simply by Talkingcited by 83
  10. 2025Process Reward Models for LLM Agents: Practical Framework and Directionscited by 81
  11. 2024Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questionscited by 70
  12. 2025SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attributioncited by 68
  13. 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 68
  14. 2025AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learningcited by 68
  15. 2025VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agentscited by 60
  16. 2025Self-Challenging Language Model Agentscited by 56
  17. 2025Horizon Reduction Makes RL Scalablecited by 55
  18. 2026Hindsight Credit Assignment for Long-Horizon LLM Agentscited by 51
  19. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 49
  20. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  21. 2024On Targeted Manipulation and Deception when Optimizing LLMs for User Feedbackcited by 45
  22. 2025ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestrationcited by 44
  23. 2025Agentic Entropy-Balanced Policy Optimizationcited by 39
  24. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Taskscited by 37
  25. 2025Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interactioncited by 34
  26. 2024DynaSaur: Large Language Agents Beyond Predefined Actionscited by 30
  27. 2024Controlling Large Language Model Agents with Entropic Activation Steeringcited by 29
  28. 2024Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHFcited by 29
  29. 2025PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifiercited by 28
  30. 2025Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMscited by 28
  31. 2026Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learningcited by 28
  32. 2025Multi-Turn Code Generation Through Single-Step Rewardscited by 27
  33. 2025Reinforce LLM Reasoning through Multi-Agent Reflectioncited by 26
  34. 2026The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Breakcited by 25
  35. 2025SWE-Dev: Building Software Engineering Agents with Training and Inference Scalingcited by 24
  36. 2024ReAct Meets ActRe: When Language Agents Enjoy Training Data Autonomycited by 22
  37. 2024Can We Rely on LLM Agents to Draft Long-Horizon Plans? Let's Take TravelPlanner as an Examplecited by 22
  38. 2024StepTool: Enhancing Multi-Step Tool Usage in LLMs via Step-Grained Reinforcement Learningcited by 22

50 most cited of 212 All 212 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcement Learning for Long-Horizon Multi-Turn Search Agents
  2. 2025Scaling LLM Multi-turn RL with End-to-end Summarization-based Context Management
  3. 2024LLM-based Multi-Agent Reinforcement Learning: Current and Future Directions
  4. 2025AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Framework
  5. 2025A Survey of Reinforcement Learning for Large Reasoning Models
  6. 2025Toward Efficient Exploration by Large Language Model Agents
  7. 2025Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agents
  8. 2026Rethinking Agentic Reinforcement Learning In Large Language Models