mlsanityData to 7 Oct 2026

On the Effects of Data Scale on UI Control Agents

arXiv 2406.03679 ↗First posted 6 Jun 2024cs.AI cs.LGcited by 187

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

Autonomous agents that control computer interfaces to accomplish human tasks are emerging. Leveraging LLMs to power such agents has been of special interest, but unless fine-tuned on human-collected task demonstrations, performance is still relatively low. In this work we study whether fine-tuning alone is a viable approach for building real-world computer control agents. In particularly, we investigate how performance measured on both high and low-level tasks in domain and out of domain scales as more training data is collected. To this end we collect and release a new dataset, AndroidControl, consisting of 15,283 demonstrations of everyday tasks with Android apps. Compared to existing datasets, each AndroidControl task instance includes both high and low-level human-generated instructions, allowing us to explore the level of task complexity an agent can handle. Moreover, AndroidControl is the most diverse computer control dataset to date, including 14,548 unique tasks over 833 Android apps, thus allowing us to conduct in-depth analysis of the model performance in and out of the domain of the training data. Using the dataset, we find that when tested in domain fine-tuned models outperform zero and few-shot baselines and scale in such a way that robust performance might feasibly be obtained simply by collecting more data. Out of domain, performance scales significantly more slowly and suggests that in particular for high-level tasks, fine-tuning on more data alone may be insufficient for achieving robust out-of-domain performance.

Key facts. Cited by 187 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 31 of its 41 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

31 of 41 references.

References · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  10. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  12. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
19 more
  1. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  2. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  3. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  4. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  5. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  6. 2023Language Models can Solve Computer Taskscited by 371
  7. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  8. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  9. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
  10. 2021Scaling Laws for Transfercited by 265
  11. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  12. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
  13. 2023Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Controlcited by 180
  14. 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 93
  15. 2022A data-driven approach for learning to control computerscited by 87
  16. 2021The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuningcited by 74
  17. 2018Learning to Navigate the Webcited by 60
  18. 2021Mobile App Tasks with Iterative Feedback (MoTIF): Addressing Task Feasibility in Interactive Visual Environmentscited by 28
  19. 2023UINav: A Practical Approach to Train On-Device Automation Agentscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 90; 2026: 91.

187 papers.

Sort:

By area:

  1. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 370
  2. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 174
  3. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  4. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesiscited by 155
  5. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
  6. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 132
  7. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 110
  8. 2025Reinforcement Learning for Self-Improving Agent with Skill Librarycited by 91
  9. 2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 90
  10. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 89
  11. 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agentscited by 88
  12. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
38 more
  1. 2025Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agentscited by 68
  2. 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 64
  3. 2025Ming-Omni: A Unified Multimodal Model for Perception and Generationcited by 62
  4. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 58
  5. 2025ZeroGUI: Automating Online GUI Learning at Zero Human Costcited by 44
  6. 2025TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agentscited by 40
  7. 2026UI-Venus-1.5 Technical Reportcited by 39
  8. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  9. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  10. 2025A Comprehensive Survey of Agents for Computer Use: Foundations, Challenges, and Future Directionscited by 33
  11. 2025UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Groundingcited by 32
  12. 2024Foundations and Recent Trends in Multimodal Mobile Agents: A Surveycited by 31
  13. 2026Code2World: A GUI World Model via Renderable Code Generationcited by 26
  14. 2025GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Taskscited by 25
  15. 2025GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behaviorcited by 22
  16. 2025UItron: Foundational GUI Agent with Advanced Perception and Planningcited by 22
  17. 2025VideoAgentTrek: Computer Use Pretraining from Unlabeled Videoscited by 22
  18. 2026MobileDreamer: Generative Sketch World Model for GUI Agentcited by 21
  19. 2025UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoningcited by 20
  20. 2025FingerTip 20K: A Benchmark for Proactive and Personalized Mobile LLM Agentscited by 19
  21. 2025MagicGUI: A Foundational Mobile GUI Agent with Scalable Data Pipeline and Reinforcement Fine-tuningcited by 19
  22. 2025Auto-scaling Continuous Memory for GUI Agentcited by 19
  23. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  24. 2026KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluationcited by 19
  25. 2025AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMscited by 18
  26. 2025OS-Sentinel: Towards Safety-Enhanced Mobile GUI Agents via Hybrid Validation in Realistic Workflowscited by 18
  27. 2025MobileWorldBench: Towards Semantic World Modeling For Mobile Agentscited by 18
  28. 2025A3: Android Agent Arena for Mobile GUI Agents with Essential-State Procedural Evaluationcited by 17
  29. 2025Efficient Agent Training for Computer Usecited by 17
  30. 2025Build the web for agents, not agents for the webcited by 17
  31. 2025Mobile-R1: Towards Interactive Capability for VLM-Based Mobile Agent via Systematic Trainingcited by 17
  32. 2026A Comprehensive Survey on Agent Skills: Taxonomy, Techniques, and Applicationscited by 17
  33. 2025ScaleTrack: Scaling and back-tracking Automated GUI Agentscited by 16
  34. 2025Mobile-Bench-v2: A More Realistic and Comprehensive Benchmark for VLM-based Mobile Agentscited by 16
  35. 2026PersonalAlign: Hierarchical Implicit Intent Alignment for Personalized GUI Agent with Long-Term User-Centric Recordscited by 16
  36. 2025Scalable Video-to-Dataset Generation for Cross-Platform Mobile Agentscited by 14
  37. 2025Retrieval-augmented GUI Agents with Generative Guidelinescited by 14
  38. 2025Ferret-UI Lite: Lessons from Building Small On-Device GUI Agentscited by 14

50 most cited of 187 All 187 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Benchmarking Mobile Device Control Agents across Diverse Configurations
  2. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents
  3. 2025AppVLM: A Lightweight Vision Language Model for Online App Control
  4. 2024SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation
  5. 2025Scaling Agents for Computer Use
  6. 2024AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents
  7. 2026Benchmarking General Mobile Assistants in Challenging Real-World Scenarios
  8. 2026Same Tasks, Different Apps: Why Mobile GUI Agents Fail to Generalize?