mlsanityData to 7 Oct 2026

Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration

arXiv 1802.08802 ↗First posted 24 Feb 2018cs.AIcited by 282

Not on the map. Why?

Cited by 199 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Reinforcement learning (RL) agents improve through trial-and-error, but when reward is sparse and the agent cannot discover successful action sequences, learning stagnates. This has been a notable problem in training deep RL agents to perform web-based tasks, such as booking flights or replying to emails, where a single mistake can ruin the entire sequence of actions. A common remedy is to "warm-start" the agent by pre-training it to mimic expert demonstrations, but this is prone to overfitting. Instead, we propose to constrain exploration using demonstrations. From each demonstration, we induce high-level "workflows" which constrain the allowable actions at each time step to be similar to those in the demonstration (e.g., "Step 1: click on a textbox; Step 2: enter some text"). Our exploration policy then learns to identify successful workflows and samples actions that satisfy these workflows. Workflows prune out bad exploration directions and accelerate the agent's ability to discover rewards. We use our approach to train a novel neural policy designed to handle the semi-structured nature of websites, and evaluate on a suite of web tasks, including the recent World of Bits benchmark. We achieve new state-of-the-art results, and show that workflow-guided exploration improves sample efficiency over behavioral cloning by more than 100x.

Key facts. Cited by 282 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 8; 2023: 25; 2024: 80; 2025: 73; 2026: 90.

282 papers.

Sort:

By area:

  1. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  2. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  3. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  4. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  5. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  6. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  7. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  8. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 551
  9. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  10. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  11. 2023Language Models can Solve Computer Taskscited by 371
  12. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
38 more
  1. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 348
  2. 2024Agent Workflow Memorycited by 342
  3. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  4. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 306
  5. 2023Tool Learning with Foundation Modelscited by 302
  6. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  7. 2023AdaPlanner: Adaptive Planning from Feedback with Language Modelscited by 193
  8. 2024OmniParser for Pure Vision Based GUI Agentcited by 193
  9. 2025Survey on Evaluation of LLM-based Agentscited by 190
  10. 2024On the Effects of Data Scale on UI Control Agentscited by 187
  11. 2023Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Controlcited by 180
  12. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 177
  13. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  14. 2025WebWalker: Benchmarking LLMs in Web Traversalcited by 171
  15. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  16. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  17. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 155
  18. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesiscited by 155
  19. 2024The BrowserGym Ecosystem for Web Agent Researchcited by 152
  20. 2024Tree Search for Language Model Agentscited by 148
  21. 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agentscited by 144
  22. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 136
  23. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
  24. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 132
  25. 2024ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agentscited by 126
  26. 2024WebCanvas: Benchmarking Web Agents in Online Environmentscited by 119
  27. 2024Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigationcited by 116
  28. 2024GUICourse: From General Vision Language Models to Versatile GUI Agentscited by 112
  29. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 110
  30. 2024AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agentscited by 110
  31. 2024GUI Agents with Foundation Models: A Comprehensive Surveycited by 110
  32. 2024AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agentscited by 105
  33. 2025Inducing Programmatic Skills for Agentic Taskscited by 103
  34. 2024ScreenAgent: A Vision Language Model-driven Computer Control Agentcited by 101
  35. 2024VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agentscited by 101
  36. 2024GUI Agents: A Surveycited by 101
  37. 2024WebPilot: A Versatile and Autonomous Multi-Agent System for Web Task Execution with Strategic Explorationcited by 100
  38. 2025AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectoriescited by 99

50 most cited of 282 All 282 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learning
  2. 2026OpAgent: Operator Agent for Web Navigation
  3. 2025Agentic Reinforcement Learning for Search Misaligns Instruction-Tuning
  4. 2025Learning from Online Videos at Inference Time for Computer-Use Agents
  5. 2026SynPlanResearch-R1: Encouraging Tool Exploration for Deep Research with Synthetic Plans
  6. 2026How to Train Your Deep Research Agent? Prompt, Reward, and Policy Optimization in Search-R1
  7. 2025WebSeer: Training Deeper Search Agents through Reinforcement Learning with Self-Reflection
  8. 2025Agentic Reinforcement Learning with Implicit Step Rewards