mlsanityData to 7 Oct 2026

WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learning

arXiv 2505.16421 ↗First posted 22 May 2025cs.CL cs.LGcited by 162

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

While reinforcement learning (RL) has demonstrated remarkable success in enhancing large language models (LLMs), it has primarily focused on single-turn tasks such as solving math problems. Training effective web agents for multi-turn interactions remains challenging due to the complexity of long-horizon decision-making across dynamic web interfaces. In this work, we present WebAgent-R1, a simple yet effective end-to-end multi-turn RL framework for training web agents. It learns directly from online interactions with web environments by asynchronously generating diverse trajectories, entirely guided by binary rewards depending on task success. Experiments on the WebArena-Lite benchmark demonstrate the effectiveness of WebAgent-R1, boosting the task success rate of Qwen-2.5-3B from 6.1% to 33.9% and Llama-3.1-8B from 8.5% to 44.8%, significantly outperforming existing state-of-the-art methods and strong proprietary models such as OpenAI o3. In-depth analyses reveal the effectiveness of the thinking-based prompting strategy and test-time scaling through increased interactions for web tasks. We further investigate different RL initialization policies by introducing two variants, namely WebAgent-R1-Zero and WebAgent-R1-CoT, which highlight the importance of the warm-up training stage (i.e., behavior cloning) and provide insights on incorporating long chain-of-thought (CoT) reasoning in web agents.

Key facts. Cited by 162 papers in the map (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 51 of its 64 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

51 of 64 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning4 Nov
  2. 2024OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimization25 Oct
  3. 2024Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation17 Oct
  4. 2024Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systems17 Jul
  5. 2024Large Language Models Can Self-Improve At Web Agent Tasks30 May
  6. 2024WILBUR: Adaptive In-Context Learning for Robust and Accurate Web Agents8 Apr
  7. 2024AutoWebGLM: A Large Language Model-based Web Navigating Agent4 Apr
  8. 2024GPT-4V(ision) is a Generalist Web Agent, if Grounded3 Jan
  9. 2023LASER: LLM Agent with State-Space Exploration for Web Navigation15 Sep
  10. 2023WebArena: A Realistic Web Environment for Building Autonomous Agents25 Jul
  11. 2023Mind2Web: Towards a Generalist Agent for the Web9 Jun
  12. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents4 Jul

Other references · 39

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  5. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  7. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  8. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  9. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  10. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  11. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  12. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
27 more
  1. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  2. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 505
  3. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  4. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  5. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  6. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
  7. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  8. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 212
  9. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 210
  10. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 203
  11. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searchingcited by 203
  12. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 201
  13. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
  14. 2022Language Models of Code are Few-Shot Commonsense Learnerscited by 160
  15. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 136
  16. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 108
  17. 2024VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agentscited by 101
  18. 2024WebPilot: A Versatile and Autonomous Multi-Agent System for Web Task Execution with Strategic Explorationcited by 100
  19. 2021AndroidEnv: A Reinforcement Learning Platform for Androidcited by 90
  20. 2023Agent Lumos: Unified and Modular Training for Open-Source Language Agentscited by 72
  21. 2024AutoGuide: Automated Generation and Selection of Context-Aware Guidelines for Large Language Model Agentscited by 69
  22. 2023SteP: Stacked LLM Policies for Web Actionscited by 66
  23. 2024Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Modelscited by 56
  24. 2024Dual-View Visual Contextualization for Web Navigationcited by 24
  25. 2022Show Me More Details: Discovering Hierarchies of Procedures from Semi-structured Web Datacited by 22
  26. 2025SelfElicit: Your Language Model Secretly Knows Where is the Relevant Evidencecited by 18
  27. 2023Take a Break in the Middle: Investigating Subgoals towards Hierarchical Script Generationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 50; 2026: 112.

162 papers.

Sort:

By area:

  1. 2025Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learningcited by 220
  2. 2025WebDancer: Towards Autonomous Information Seeking Agencycited by 180
  3. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  4. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 158
  5. 2025Agent Learning via Early Experiencecited by 79
  6. 2025One Token to Fool LLM-as-a-Judgecited by 56
  7. 2025MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environmentcited by 51
  8. 2025From Web Search towards Agentic Deep Research: Incentivizing Search with Reasoning Agentscited by 49
  9. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  10. 2025Agentic Reinforcement Learning with Implicit Step Rewardscited by 45
  11. 2025Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agentscited by 43
  12. 2025Scaling Agent Learning via Experience Synthesiscited by 40
38 more
  1. 2025From Word to World: Can Large Language Models be Implicit Text-based World Models?cited by 40
  2. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 40
  3. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Taskscited by 37
  4. 2025SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agentscited by 29
  5. 2025ARE: Scaling Up Agent Environments and Evaluationscited by 28
  6. 2025Atom-Searcher: Enhancing Agentic Deep Research via Fine-Grained Atomic Thought Rewardcited by 25
  7. 2026WebXSkill: Skill Learning for Autonomous Web Agentscited by 24
  8. 2026ExpSeek: Self-Triggered Experience Seeking for Web Agentscited by 23
  9. 2025UItron: Foundational GUI Agent with Advanced Perception and Planningcited by 22
  10. 2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applicationscited by 21
  11. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  12. 2026From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanismscited by 19
  13. 2025WebSynthesis: World-Model-Guided MCTS for Efficient WebUI-Trajectory Synthesiscited by 18
  14. 2025Customer-R1: Personalized Simulation of Human Behaviors via RL-based LLM Agent in Online Shoppingcited by 16
  15. 2025WebCoach: Self-Evolving Web Agents with Cross-Session Memory Guidancecited by 15
  16. 2025WebCoT: Enhancing Web Agent Reasoning by Reconstructing Chain-of-Thought in Reflection, Branching, and Rollbackcited by 14
  17. 2026The Confidence Dichotomy: Analyzing and Mitigating Miscalibration in Tool-Use Agentscited by 14
  18. 2025DraCo: Draft as CoT for Text-to-Image Preview and Rare Concept Generationcited by 13
  19. 2026Verified Critical Step Optimization for LLM Agentscited by 13
  20. 2026OpenEarthAgent: A Unified Framework for Tool-Augmented Geospatial Agentscited by 13
  21. 2026ReSkill: Reconciling Skill Creation with Policy Optimization in Agentic RLcited by 13
  22. 2025SynthAgent: Adapting Web Agents with Synthetic Supervisioncited by 12
  23. 2025GTM: Simulating the World of Tools for AI Agentscited by 12
  24. 2025AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agentscited by 12
  25. 2026DynaWeb: Model-Based Reinforcement Learning of Web Agentscited by 12
  26. 2026When Users Change Their Mind: Evaluating Interruptible Agents in Long-Horizon Web Navigationcited by 12
  27. 2025WebGen-Agent: Enhancing Interactive Website Generation with Multi-Level Feedback and Step-Level Reinforcement Learningcited by 11
  28. 2025Autonomous Agents for Scientific Discovery: Orchestrating Scientists, Language, Code, and Physicscited by 11
  29. 2025Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberationcited by 11
  30. 2026UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experiencecited by 11
  31. 2026Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RLcited by 11
  32. 2026SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillationcited by 11
  33. 2025Rank-GRPO: Training LLM-based Conversational Recommender Systems with Reinforcement Learningcited by 10
  34. 2025OpenApps: Simulating Environment Variations to Measure UI-Agent Reliabilitycited by 10
  35. 2026A Subgoal-driven Framework for Improving Long-Horizon LLM Agentscited by 10
  36. 2026GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agentscited by 9
  37. 2026Memory-R2: Fair Credit Assignment for Long-Horizon Memory-Augmented LLM Agentscited by 9
  38. 2026From Question Answering to Task Completion: A Survey on Agent System and Harness Designcited by 9

50 most cited of 162 All 162 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026WebGym: Scaling Training Environments for Visual Web Agents with Realistic Tasks
  2. 2025How to Train Your LLM Web Agent: A Statistical Diagnosis
  3. 2025WorkForceAgent-R1: Incentivizing Reasoning Capability in LLM-based Web Agents via Reinforcement Learning
  4. 2025AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Framework
  5. 2025A Practitioner's Guide to Multi-turn Agentic Reinforcement Learning
  6. 2025AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning
  7. 2025WebExplorer: Explore and Evolve for Training Long-Horizon Web Agents
  8. 2026InfiniteWeb: Scalable Web Environment Synthesis for GUI Agent Training