mlsanityData to 7 Oct 2026

AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents

arXiv 2410.24024 ↗First posted 31 Oct 2024cs.AIcited by 105

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

Autonomous agents have become increasingly important for interacting with the real world. Android agents, in particular, have been recently a frequently-mentioned interaction method. However, existing studies for training and evaluating Android agents lack systematic research on both open-source and closed-source models. In this work, we propose AndroidLab as a systematic Android agent framework. It includes an operation environment with different modalities, action space, and a reproducible benchmark. It supports both large language models (LLMs) and multimodal models (LMMs) in the same action space. AndroidLab benchmark includes predefined Android virtual devices and 138 tasks across nine apps built on these devices. By using the AndroidLab environment, we develop an Android Instruction dataset and train six open-source LLMs and LMMs, lifting the average success rates from 4.59% to 21.50% for LLMs and from 1.93% to 13.28% for LMMs. AndroidLab is open-sourced and publicly available at https://github.com/THUDM/Android-Lab.

Key facts. Cited by 105 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 28 of its 58 references that are in the map. First posted on arXiv 31 Oct 2024.

Builds on

28 of 58 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Benchmarking Mobile Device Control Agents across Diverse Configurations25 Apr
  2. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments11 Apr
  3. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web27 Feb
  4. 2024Understanding the Weakness of Large Language Model Agents within a Complex Android Environment9 Feb
  5. 2023You Only Look at Screens: Multimodal Chain-of-Action Agents20 Sep

Other references · 23

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  7. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  8. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  9. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  10. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  11. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  12. 2018Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfacescited by 376
11 more
  1. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  2. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  3. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
  4. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
  5. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  6. 2024VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agentscited by 101
  7. 2023WebGLM: Towards An Efficient Web-Enhanced Question Answering System with Human Preferencescited by 100
  8. 2021AndroidEnv: A Reinforcement Learning Platform for Androidcited by 90
  9. 2022META-GUI: Towards Multi-modal Conversational Agents on Mobile GUIcited by 81
  10. 2021Mobile App Tasks with Iterative Feedback (MoTIF): Addressing Task Feasibility in Interactive Visual Environmentscited by 28
  11. 2021Revisiting, Benchmarking and Exploring API Recommendation: How Far Are We?cited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 47; 2026: 56.

105 papers.

Sort:

By area:

  1. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  2. 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agentscited by 88
  3. 2025MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environmentscited by 69
  4. 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 64
  5. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  6. 2025AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Frameworkcited by 58
  7. 2025DataSciBench: An LLM Agent Benchmark for Data Sciencecited by 53
  8. 2025ZeroGUI: Automating Online GUI Learning at Zero Human Costcited by 44
  9. 2025MobileRL: Online Agentic Reinforcement Learning for Mobile GUI Agentscited by 43
  10. 2026Code as Agent Harnesscited by 40
  11. 2026UI-Venus-1.5 Technical Reportcited by 39
  12. 2025Advancing Mobile GUI Agents: A Verifier-Driven Approach to Practical Deploymentcited by 38
38 more
  1. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  2. 2025UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agentscited by 33
  3. 2024Foundations and Recent Trends in Multimodal Mobile Agents: A Surveycited by 31
  4. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  5. 2026OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Taskscited by 29
  6. 2025GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Taskscited by 25
  7. 2026AgentOCR: Reimagining Agent History via Optical Self-Compressioncited by 22
  8. 2025FingerTip 20K: A Benchmark for Proactive and Personalized Mobile LLM Agentscited by 19
  9. 2026KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluationcited by 19
  10. 2025A3: Android Agent Arena for Mobile GUI Agents with Essential-State Procedural Evaluationcited by 17
  11. 2025Mobile-Bench-v2: A More Realistic and Comprehensive Benchmark for VLM-based Mobile Agentscited by 16
  12. 2025ProBench: Benchmarking GUI Agents with Accurate Process Informationcited by 14
  13. 2026Don't Act Blindly: Robust GUI Automation via Action-Effect Verification and Self-Correctioncited by 14
  14. 2025EchoTrail-GUI: Building Actionable Memory for GUI Agents via Critic-Guided Self-Explorationcited by 13
  15. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 12
  16. 2025Does Chain-of-Thought Reasoning Help Mobile GUI Agent? An Empirical Studycited by 11
  17. 2025Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling Systemcited by 11
  18. 2025Quick on the Uptake: Eliciting Implicit Intents from Human Demonstrations for Personalized Mobile-Use Agentscited by 11
  19. 2026UI-Mem: Self-Evolving Experience Memory for Online Reinforcement Learning in Mobile GUI Agentscited by 11
  20. 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestrationcited by 11
  21. 2025ProRe: A Proactive Reward System for GUI Agents via Reasoner-Actor Collaborationcited by 10
  22. 2025AgentProg: Empowering Long-Horizon GUI Agents with Program-Guided Context Managementcited by 10
  23. 2026AndroTMem: From Interaction Trajectories to Anchored Memory in Long-Horizon GUI Agentscited by 10
  24. 2026OpenMobile: Building Open Mobile Agents with Task and Trajectory Synthesiscited by 10
  25. 2025AutoEval: A Practical Framework for Autonomous Evaluation of Mobile Agentscited by 9
  26. 2025MobileUse: A GUI Agent with Hierarchical Reflection for Autonomous Mobile Operationcited by 9
  27. 2025ColorBench: Benchmarking Mobile Agents with Graph-Structured Framework for Complex Long-Horizon Taskscited by 9
  28. 2026MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Researchcited by 9
  29. 2025MAS-Bench: A Unified Benchmark for Shortcut-Augmented Hybrid Mobile GUI Agentscited by 8
  30. 2025ProSoftArena: Benchmarking Hierarchical Capabilities of Multimodal Agents in Professional Software Environmentscited by 7
  31. 2026AmbiBench: Benchmarking Mobile GUI Agents Beyond One-Shot Instructions in the Wildcited by 7
  32. 2025EcoAgent: An Efficient Device-Cloud Collaborative Multi-Agent Framework for Mobile Automationcited by 6
  33. 2025TextAtari: 100K Frames Game Playing with Language Agentscited by 6
  34. 2025InquireMobile: Teaching VLM-based Mobile Agent to Request Human Assistance via Reinforcement Fine-Tuningcited by 6
  35. 2025ColorAgent: Building A Robust, Personalized, and Interactive OS Agentcited by 6
  36. 2026Generalization in Online Reinforcement Learning for Mobile Agentscited by 6
  37. 2026Android Coach: Improve Online Agentic Training Efficiency with Single State Multiple Actionscited by 6
  38. 2026iOSWorld: A Benchmark for Personally Intelligent Phone Agentscited by 6

50 most cited of 105 All 105 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation
  2. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents
  3. 2024MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents
  4. 2026Benchmarking General Mobile Assistants in Challenging Real-World Scenarios
  5. 2025AndroidGen: Building an Android Language Agent under Data Scarcity
  6. 2024Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agents
  7. 2024CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents
  8. 2024On the Effects of Data Scale on UI Control Agents