mlsanityData to 7 Oct 2026

A data-driven approach for learning to control computers

arXiv 2202.08137 ↗First posted 16 Feb 2022cs.LGcited by 87

Not on the map. Why?

Cited by 45 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

It would be useful for machines to use computers as humans do so that they can aid us in everyday tasks. This is a setting in which there is also the potential to leverage large-scale expert demonstrations and human judgements of interactive behaviour, which are two ingredients that have driven much recent success in AI. Here we investigate the setting of computer control using keyboard and mouse, with goals specified via natural language. Instead of focusing on hand-designed curricula and specialized action spaces, we focus on developing a scalable method centered on reinforcement learning combined with behavioural priors informed by actual human-computer interactions. We achieve state-of-the-art and human-level mean performance across all tasks within the MiniWob++ benchmark, a challenging suite of computer control problems, and find strong evidence of cross-task transfer. These results demonstrate the usefulness of a unified human-agent interface when training machines to use computers. Altogether our results suggest a formula for achieving competency beyond MiniWob++ and towards controlling computers, in general, as a human would.

Key facts. Cited by 87 papers in the map (data through 7 Oct 2026). Builds on 24 of its 32 references that are in the map. First posted on arXiv 16 Feb 2022.

Builds on

24 of 32 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  11. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  12. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
12 more
  1. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  2. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
  3. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
  4. 2021AndroidEnv: A Reinforcement Learning Platform for Androidcited by 90
  5. 2019V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Controlcited by 85
  6. 2018Learning to Navigate the Webcited by 60
  7. 2016Dialogue Learning With Human-In-The-Loopcited by 46
  8. 2021Creating Multimodal Interactive Agents with Imitation and Self-Supervised Learningcited by 40
  9. 2020How to Motivate Your Dragon: Teaching Goal-Driven Agents to Speak and Act in Fantasy Worldscited by 35
  10. 2022Environment Generation for Zero-Shot Compositional Reinforcement Learningcited by 32
  11. 2019DOM-Q-NET: Grounded RL on Structured Languagecited by 25
  12. 2021AppBuddy: Learning to Accomplish Tasks in Mobile Apps via Reinforcement Learningcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 19; 2024: 36; 2025: 15; 2026: 10.

87 papers.

Sort:

By area:

  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  3. 2023Teaching Large Language Models to Self-Debugcited by 963
  4. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  5. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  6. 2023Augmented Language Models: a Surveycited by 398
  7. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  8. 2023Language Models can Solve Computer Taskscited by 371
  9. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  10. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 348
  11. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  12. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
38 more
  1. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 272
  2. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  3. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 246
  4. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  5. 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scalecited by 212
  6. 2023AdaPlanner: Adaptive Planning from Feedback with Language Modelscited by 193
  7. 2024On the Effects of Data Scale on UI Control Agentscited by 187
  8. 2024Agent S: An Open Agentic Framework that Uses Computers Like a Humancited by 182
  9. 2023Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Controlcited by 180
  10. 2023AutoDroid: LLM-powered Task Automation in Androidcited by 178
  11. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 177
  12. 2025WebWalker: Benchmarking LLMs in Web Traversalcited by 171
  13. 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 165
  14. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  15. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 156
  16. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 155
  17. 2024The BrowserGym Ecosystem for Web Agent Researchcited by 152
  18. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 136
  19. 2024GUICourse: From General Vision Language Models to Versatile GUI Agentscited by 112
  20. 2024AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agentscited by 110
  21. 2024GUI Agents with Foundation Models: A Comprehensive Surveycited by 110
  22. 2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 90
  23. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
  24. 2024DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agentscited by 68
  25. 2023From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfacescited by 67
  26. 2024Scaling Instructable Agents Across Many Simulated Worldscited by 67
  27. 2023SteP: Stacked LLM Policies for Web Actionscited by 66
  28. 2025Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agentscited by 66
  29. 2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 63
  30. 2022A Generalist Neural Algorithmic Learnercited by 58
  31. 2024NNetNav: Unsupervised Learning of Browser Agents Through Environment Interaction in the Wildcited by 55
  32. 2024Benchmarking Mobile Device Control Agents across Diverse Configurationscited by 54
  33. 2024Offline Training of Language Model Agents with Functions as Learnable Weightscited by 47
  34. 2023ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automationcited by 45
  35. 2024AgentStudio: A Toolkit for Building General Virtual Agentscited by 45
  36. 2024Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scalecited by 43
  37. 2025TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agentscited by 40
  38. 2023Mobile-Env: Building Qualified Evaluation Benchmarks for LLM-GUI Interactioncited by 35

50 most cited of 87 All 87 (JSON)