mlsanityData to 14 Oct 2026

Android in the Wild: A Large-Scale Dataset for Android Device Control

arXiv 2307.10088 ↗First posted 19 Jul 2023cs.LG cs.CL cs.HCcited by 319

Not on the map. Why?

Cited by 230 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

There is a growing interest in device-control systems that can interpret human natural language instructions and execute them on a digital device by directly controlling its user interface. We present a dataset for device-control research, Android in the Wild (AITW), which is orders of magnitude larger than current datasets. The dataset contains human demonstrations of device interactions, including the screens and actions, and corresponding natural language instructions. It consists of 715k episodes spanning 30k unique instructions, four versions of Android (v10-13),and eight device types (Pixel 2 XL to Pixel 6) with varying screen resolutions. It contains multi-step tasks that require semantic understanding of language and visual context. This dataset poses a new challenge: actions available through the user interface must be inferred from their visual appearance. And, instead of simple UI element-based actions, the action space consists of precise gestures (e.g., horizontal scrolls to operate carousel widgets). We organize our dataset to encourage robustness analysis of device-control systems, i.e., how well a system performs in the presence of new task descriptions, new applications, or new platform versions. We develop two agents and report performance across the dataset. The dataset is available at https://github.com/google-research/google-research/tree/master/android_in_the_wild.

Key facts. Cited by 319 papers in the map (data through 14 Oct 2026). Builds on 31 of its 50 references that are in the map. First posted on arXiv 19 Jul 2023.

Builds on

31 of 50 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  7. 2023Language Models can Solve Computer Taskscited by 372
  8. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 366
  9. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 284
  10. 2022Interactive Language: Talking to Robots in Real Timecited by 218
  11. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 209
  12. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 181
19 more
  1. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 160
  2. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  3. 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 95
  4. 2021AndroidEnv: A Reinforcement Learning Platform for Androidcited by 90
  5. 2022A data-driven approach for learning to control computerscited by 88
  6. 2017Weakly Supervised Action Learning with RNN based Fine-to-coarse Modelingcited by 70
  7. 2022Enabling Conversational Interaction with Mobile UI using Large Language Modelscited by 66
  8. 2018Learning to Navigate the Webcited by 60
  9. 2021Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixelscited by 58
  10. 2020ActionBert: Leveraging User Actions for Semantic Understanding of User Interfacescited by 43
  11. 2022Environment Generation for Zero-Shot Compositional Reinforcement Learningcited by 32
  12. 2020Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination?cited by 29
  13. 2021Mobile App Tasks with Iterative Feedback (MoTIF): Addressing Task Feasibility in Interactive Visual Environmentscited by 28
  14. 2019DOM-Q-NET: Grounded RL on Structured Languagecited by 25
  15. 2020Unblind Your Apps: Predicting Natural-Language Labels for Mobile GUI Components by Deep Learningcited by 25
  16. 2022UGIF: UI Grounded Instruction Followingcited by 25
  17. 2022Towards Better Semantic Understanding of Mobile Interfacescited by 24
  18. 2021AppBuddy: Learning to Accomplish Tasks in Mobile Apps via Reinforcement Learningcited by 17
  19. 2023Lexi: Self-Supervised Learning of the UI Languagecited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 11; 2024: 70; 2025: 122; 2026: 116.

319 papers.

Sort:

By area:

  1. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  2. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
  3. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 567
  4. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 497
  5. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
  6. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 415
  7. 2024Agent Workflow Memorycited by 383
  8. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 382
  9. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 363
  10. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 278
  11. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  12. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 256
38 more
  1. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 232
  2. 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scalecited by 223
  3. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 214
  4. 2025Plan-and-Act: Improving Planning of Agents for Long-Horizon Taskscited by 195
  5. 2024On the Effects of Data Scale on UI Control Agentscited by 194
  6. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 194
  7. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  8. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 180
  9. 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learningcited by 168
  10. 2024The BrowserGym Ecosystem for Web Agent Researchcited by 161
  11. 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 160
  12. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agentscited by 159
  13. 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agentscited by 159
  14. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 157
  15. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 139
  16. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  17. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 137
  18. 2024AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorialscited by 130
  19. 2024WebCanvas: Benchmarking Web Agents in Online Environmentscited by 121
  20. 2024Dissecting Adversarial Robustness of Multimodal LM Agentscited by 120
  21. 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 117
  22. 2024GUICourse: From General Vision Language Models to Versatile GUI Agentscited by 114
  23. 2024GUI Agents with Foundation Models: A Comprehensive Surveycited by 111
  24. 2024AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agentscited by 107
  25. 2024VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agentscited by 103
  26. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 101
  27. 2024AutoGLM: Autonomous Foundation Agents for GUIscited by 100
  28. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 98
  29. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 95
  30. 2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 93
  31. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
  32. 2024SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluationcited by 81
  33. 2024Beyond Browsing: API-Based Web Agentscited by 78
  34. 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 77
  35. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
  36. 2025MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environmentscited by 72
  37. 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 69
  38. 2024DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agentscited by 68

50 most cited of 319 All 319 (JSON)