mlsanityData to 14 Oct 2026

WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning

arXiv 2411.02337 ↗First posted 4 Nov 2024cs.CLcited by 256

On the map: LLM agents and tool use → Web, search, and research agents

also in 1 more
Abstract (arXiv)

Large language models (LLMs) have shown remarkable potential as autonomous agents, particularly in web-based tasks. However, existing LLM web agents heavily rely on expensive proprietary LLM APIs, while open LLMs lack the necessary decision-making capabilities. This paper introduces WebRL, a self-evolving online curriculum reinforcement learning framework designed to train high-performance web agents using open LLMs. WebRL addresses three key challenges in building LLM web agents, including the scarcity of training tasks, sparse feedback signals, and policy distribution drift in online learning. Specifically, WebRL incorporates 1) a self-evolving curriculum that generates new tasks from unsuccessful attempts, 2) a robust outcome-supervised reward model (ORM), and 3) adaptive reinforcement learning strategies to ensure consistent improvements. We apply WebRL to transform open Llama-3.1 and GLM-4 models into proficient web agents. On WebArena-Lite, WebRL improves the success rate of Llama-3.1-8B from 4.8% to 42.4%, and from 6.1% to 43% for GLM-4-9B. These open models significantly surpass the performance of GPT-4-Turbo (17.6%) and GPT-4o (13.9%) and outperform previous state-of-the-art web agents trained on open LLMs (AutoWebGLM, 18.2%). Our findings demonstrate WebRL's effectiveness in bridging the gap between open and proprietary LLM-based web agents, paving the way for more accessible and powerful autonomous web interaction systems.

Key facts. Cited by 256 papers in the map (data through 14 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 51 of its 53 references that are in the map. First posted on arXiv 4 Nov 2024.

Builds on

51 of 53 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Agent Workflow Memory11 Sep
  2. 2024AutoWebGLM: A Large Language Model-based Web Navigating Agent4 Apr
  3. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models25 Jan
  4. 2024GPT-4V(ision) is a Generalist Web Agent, if Grounded3 Jan
  5. 2023WebArena: A Realistic Web Environment for Building Autonomous Agents25 Jul
  6. 2023Mind2Web: Towards a Generalist Agent for the Web9 Jun
  7. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents4 Jul

Other references · 44

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  7. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  8. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  9. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  10. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  11. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  12. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
32 more
  1. 2023Reasoning with Language Model is Planning with World Modelcited by 861
  2. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  4. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
  5. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  6. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 415
  7. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353
  8. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 338
  9. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 319
  10. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 301
  11. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 282
  12. 2023FireAct: Toward Language Agent Fine-tuningcited by 256
  13. 2024Iterative Reasoning Preference Optimizationcited by 246
  14. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  15. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 228
  16. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 214
  17. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 208
  18. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvementcited by 205
  19. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  20. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agentscited by 159
  21. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  22. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 148
  23. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 144
  24. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  25. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 137
  26. 2021Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observabilitycited by 109
  27. 2024VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agentscited by 103
  28. 2024WebPilot: A Versatile and Autonomous Multi-Agent System for Web Task Execution with Strategic Explorationcited by 101
  29. 2023BOLAA: Benchmarking and Orchestrating LLM-augmented Autonomous Agentscited by 84
  30. 2024AgentOhana: Design Unified Data and Training Pipeline for Effective Agent Learningcited by 53
  31. 2024Middleware for LLMs: Tools Are Instrumental for Language Agents in Complex Environmentscited by 39
  32. 2023LLM As DBAcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 102; 2026: 151.

256 papers.

Sort:

By area:

  1. 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agentscited by 238
  2. 2025Plan-and-Act: Improving Planning of Agents for Long-Horizon Taskscited by 195
  3. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  4. 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 174
  5. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 170
  6. 2025WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learningcited by 165
  7. 2024Large Language Model-Brained GUI Agents: A Surveycited by 162
  8. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 139
  9. 2025OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automationcited by 131
  10. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  11. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  12. 2025Self-Evolving Curriculum for LLM Reasoningcited by 112
38 more
  1. 2025Inducing Programmatic Skills for Agentic Taskscited by 111
  2. 2025Agentic Large Language Models, a surveycited by 108
  3. 2024GUI Agents: A Surveycited by 103
  4. 2025Reinforcement Learning for Self-Improving Agent with Skill Librarycited by 100
  5. 2026OpenClaw-RL: Train Any Agent Simply by Talkingcited by 90
  6. 2025Agent Learning via Early Experiencecited by 85
  7. 2025AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learningcited by 72
  8. 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 69
  9. 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflectioncited by 67
  10. 2025AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Frameworkcited by 62
  11. 2025ML-Master: Towards AI-for-AI via Integration of Exploration and Reasoningcited by 61
  12. 2025Self-Challenging Language Model Agentscited by 59
  13. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  14. 2025Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agentscited by 48
  15. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 46
  16. 2025ZeroGUI: Automating Online GUI Learning at Zero Human Costcited by 44
  17. 2025Scaling Agent Learning via Experience Synthesiscited by 44
  18. 2025Agent-RLVR: Training Software Engineering Agents via Guidance and Environment Rewardscited by 42
  19. 2025Web-Shepherd: Advancing PRMs for Reinforcing Web Agentscited by 41
  20. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 40
  21. 2025Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interactioncited by 34
  22. 2025Search Self-play: Pushing the Frontier of Agent Capability without Supervisioncited by 34
  23. 2025Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Modelscited by 30
  24. 2025WebChoreArena: Evaluating Web Browsing Agents on Realistic Tedious Web Taskscited by 30
  25. 2025Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMscited by 30
  26. 2026Gym-Anything: Turn any Software into an Agent Environmentcited by 27
  27. 2025SWE-Dev: Building Software Engineering Agents with Training and Inference Scalingcited by 26
  28. 2025OS-Kairos: Adaptive Interaction for MLLM-Powered GUI Agentscited by 25
  29. 2025Atom-Searcher: Enhancing Agentic Deep Research via Fine-Grained Atomic Thought Rewardcited by 25
  30. 2025Symbiotic Cooperation for Web Agents: Harnessing Complementary Strengths of Large and Small LLMscited by 24
  31. 2025Procedural Environment Generation for Tool-Use Agentscited by 24
  32. 2026AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Conditioncited by 23
  33. 2025ProgRM: Build Better GUI Agents with Progress Rewardscited by 21
  34. 2025WebSynthesis: World-Model-Guided MCTS for Efficient WebUI-Trajectory Synthesiscited by 21
  35. 2025WebGuard: Building a Generalizable Guardrail for Web Agentscited by 21
  36. 2026From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanismscited by 21
  37. 2026SOD: Step-wise On-policy Distillation for Small Language Model Agentscited by 21
  38. 2025VEM: Environment-Free Exploration for Training GUI Agent with Value Environment Modelcited by 19

50 most cited of 256 All 256 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024SAC-GLAM: Improving Online RL for LLM agents with Soft Actor-Critic and Hindsight Relabeling
  2. 2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Data
  3. 2025Multi-Agent Evolve: LLM Self-Improve through Co-evolution
  4. 2026SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning
  5. 2026Rethinking Agentic Reinforcement Learning In Large Language Models
  6. 2026LLM-ALSO: LLM-Driven Adaptive Learning-Signal Optimization for Multi-Agent Reinforcement Learning
  7. 2023Adapting LLM Agents with Universal Feedback in Communication
  8. 2025WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Model