mlsanityData to 7 Oct 2026

AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents

arXiv 2405.14573 ↗First posted 23 May 2024cs.AI cs.LGcited by 450

Not on the map. Why?

Cited by 360 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Autonomous agents that execute human tasks by controlling computers can enhance human productivity and application accessibility. However, progress in this field will be driven by realistic and reproducible benchmarks. We present AndroidWorld, a fully functional Android environment that provides reward signals for 116 programmatic tasks across 20 real-world Android apps. Unlike existing interactive environments, which provide a static test set, AndroidWorld dynamically constructs tasks that are parameterized and expressed in natural language in unlimited ways, thus enabling testing on a much larger and more realistic suite of tasks. To ensure reproducibility, each task includes dedicated initialization, success-checking, and tear-down logic, which modifies and inspects the device's system state. We experiment with baseline agents to test AndroidWorld and provide initial results on the benchmark. Our best agent can complete 30.6% of AndroidWorld's tasks, leaving ample room for future work. Furthermore, we adapt a popular desktop web agent to work on Android, which we find to be less effective on mobile, suggesting future research is needed to achieve universal, cross-platform agents. Finally, we also conduct a robustness analysis, showing that task variations can significantly affect agent performance, demonstrating that without such testing, agent performance metrics may not fully reflect practical challenges. AndroidWorld and the experiments in this paper are available at github.com/google-research/android_world.

Key facts. Cited by 450 papers in the map (data through 7 Oct 2026). Builds on 62 of its 94 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

62 of 94 references.

References · 62

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  6. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  7. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  8. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  9. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  10. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  11. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  12. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
38 more
  1. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  2. 2018DeepMind Control Suitecited by 935
  3. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  4. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  5. 2017Deep Reinforcement Learning that Matterscited by 727
  6. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  7. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
  8. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  9. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  10. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 514
  11. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 424
  12. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  13. 2023Language Models can Solve Computer Taskscited by 371
  14. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 348
  15. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  16. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
  17. 2023Challenges and Applications of Large Language Modelscited by 251
  18. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvementcited by 203
  19. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
  20. 2023Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Controlcited by 180
  21. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 177
  22. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  23. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  24. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 156
  25. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agentscited by 156
  26. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 147
  27. 2023Vision-Language Models as Success Detectorscited by 138
  28. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  29. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 136
  30. 2023On the Tool Manipulation Capability of Open-source Large Language Modelscited by 122
  31. 2023Evaluating Language-Model Agents on Realistic Autonomous Taskscited by 94
  32. 2021AndroidEnv: A Reinforcement Learning Platform for Androidcited by 90
  33. 2022A data-driven approach for learning to control computerscited by 87
  34. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
  35. 2023From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfacescited by 67
  36. 2022Enabling Conversational Interaction with Mobile UI using Large Language Modelscited by 66
  37. 2024Understanding the Weakness of Large Language Model Agents within a Complex Android Environmentcited by 60
  38. 2018How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experimentscited by 56

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 29; 2025: 150; 2026: 271.

450 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2025Qwen3-VL Technical Reportcited by 2,534
  3. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  4. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  5. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  6. 2024Agent Workflow Memorycited by 342
  7. 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scalecited by 212
  8. 2024Agent S: An Open Agentic Framework that Uses Computers Like a Humancited by 182
  9. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  10. 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agentscited by 156
  11. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 155
  12. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesiscited by 155
38 more
  1. 2024Tree Search for Language Model Agentscited by 148
  2. 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasonerscited by 140
  3. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
  4. 2025Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Taskscited by 128
  5. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  6. 2026SoK: Agentic Skills -- Beyond Tool Use in LLM Agentscited by 120
  7. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 110
  8. 2024GUI Agents with Foundation Models: A Comprehensive Surveycited by 110
  9. 2025GTA1: GUI Test-time Scaling Agentcited by 102
  10. 2024AutoGLM: Autonomous Foundation Agents for GUIscited by 96
  11. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  12. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 89
  13. 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agentscited by 88
  14. 2024SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluationcited by 79
  15. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  16. 2026Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agentscited by 72
  17. 2025MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environmentscited by 69
  18. 2024DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agentscited by 68
  19. 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 68
  20. 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflectioncited by 67
  21. 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 64
  22. 2024MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agentscited by 63
  23. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  24. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 58
  25. 2024TinyAgent: Function Calling at the Edgecited by 55
  26. 2024Ferret-UI 2: Mastering Universal User Interface Understanding Across Platformscited by 51
  27. 2025MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environmentcited by 51
  28. 2024MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device Controlcited by 49
  29. 2025ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflowscited by 47
  30. 2025AppAgentX: Evolving GUI Agents as Proficient Smartphone Userscited by 44
  31. 2025ZeroGUI: Automating Online GUI Learning at Zero Human Costcited by 44
  32. 2025AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuningcited by 44
  33. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Livecited by 43
  34. 2025TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agentscited by 40
  35. 2026Code as Agent Harnesscited by 40
  36. 2025Advancing Mobile GUI Agents: A Verifier-Driven Approach to Practical Deploymentcited by 38
  37. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  38. 2025Step-GUI Technical Reportcited by 38

50 most cited of 450 All 450 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents
  2. 2026Benchmarking General Mobile Assistants in Challenging Real-World Scenarios
  3. 2026Same Tasks, Different Apps: Why Mobile GUI Agents Fail to Generalize?
  4. 2025MobileUse: A GUI Agent with Hierarchical Reflection for Autonomous Mobile Operation
  5. 2024AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents
  6. 2025AndroidControl-Curated: Revealing the True Potential of GUI Agents through Benchmark Purification
  7. 2024On the Effects of Data Scale on UI Control Agents
  8. 2026PalmClaw: A Native On-Device Agent Framework for Mobile Phones