mlsanityData to 7 Oct 2026

MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents

arXiv 2406.08184 ↗First posted 12 Jun 2024cs.AI cs.HCcited by 63

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

Large language model (LLM)-based mobile agents are increasingly popular due to their capability to interact directly with mobile phone Graphic User Interfaces (GUIs) and their potential to autonomously manage daily tasks. Despite their promising prospects in both academic and industrial sectors, little research has focused on benchmarking the performance of existing mobile agents, due to the inexhaustible states of apps and the vague definition of feasible action sequences. To address this challenge, we propose an efficient and user-friendly benchmark, MobileAgentBench, designed to alleviate the burden of extensive manual testing. We initially define 100 tasks across 10 open-source apps, categorized by multiple levels of difficulty. Subsequently, we evaluate several existing mobile agents, including AppAgent and MobileAgent, to thoroughly and systematically compare their performance. All materials are accessible on our project webpage: https://MobileAgentBench.github.io, contributing to the advancement of both academic and industrial fields.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 26 of its 36 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

26 of 36 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Benchmarking Mobile Device Control Agents across Diverse Configurations25 Apr
  2. 2024Understanding the Weakness of Large Language Model Agents within a Complex Android Environment9 Feb
  3. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception29 Jan
  4. 2023AppAgent: Multimodal Agents as Smartphone Users21 Dec
  5. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec

Other references · 21

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  5. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  6. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  7. 2023A Survey on Multimodal Large Language Modelscited by 898
  8. 2022A Generalist Agentcited by 802
  9. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  10. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  11. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  12. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
9 more
  1. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  2. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  3. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  4. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  5. 2021AndroidEnv: A Reinforcement Learning Platform for Androidcited by 90
  6. 2018Learning to Navigate the Webcited by 60
  7. 2024Octopus v2: On-device language model for super agentcited by 45
  8. 2022Multi-Granularity Prediction for Scene Text Recognitioncited by 36
  9. 2021AppBuddy: Learning to Accomplish Tasks in Mobile Apps via Reinforcement Learningcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 33; 2026: 21.

63 papers.

Sort:

By area:

  1. 2025DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agentscited by 236
  2. 2024AI Agents That Mattercited by 199
  3. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  4. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 145
  5. 2024SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluationcited by 79
  6. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  7. 2024Ferret-UI 2: Mastering Universal User Interface Understanding Across Platformscited by 51
  8. 2025MobileRL: Online Agentic Reinforcement Learning for Mobile GUI Agentscited by 43
  9. 2024CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agentscited by 38
  10. 2025Advancing Mobile GUI Agents: A Verifier-Driven Approach to Practical Deploymentcited by 38
  11. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  12. 2024Foundations and Recent Trends in Multimodal Mobile Agents: A Surveycited by 31
38 more
  1. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  2. 2025MobileSteward: Integrating Multiple App-Oriented Agents with Self-Evolution to Automate Cross-App Instructionscited by 16
  3. 2025Mobile-Bench-v2: A More Realistic and Comprehensive Benchmark for VLM-based Mobile Agentscited by 16
  4. 2024Attention-driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models without Fine-Tuningcited by 13
  5. 2025Agent-SAMA: State-Aware Mobile Assistantcited by 12
  6. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 12
  7. 2025Advancing Language Multi-Agent Learning with Credit Re-Assignment for Interactive Environment Generalizationcited by 11
  8. 2025Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling Systemcited by 11
  9. 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestrationcited by 11
  10. 2025MobileA3gent: Training Mobile GUI Agents Using Decentralized Self-Sourced Data from Diverse Userscited by 10
  11. 2025From Assistants to Adversaries: Exploring the Security Risks of Mobile LLM Agentscited by 10
  12. 2025ProRe: A Proactive Reward System for GUI Agents via Reasoner-Actor Collaborationcited by 10
  13. 2025AgentProg: Empowering Long-Horizon GUI Agents with Program-Guided Context Managementcited by 10
  14. 2026AndroTMem: From Interaction Trajectories to Anchored Memory in Long-Horizon GUI Agentscited by 10
  15. 2025ColorBench: Benchmarking Mobile Agents with Graph-Structured Framework for Complex Long-Horizon Taskscited by 9
  16. 2025What Limits Virtual Agent Application? OmniBench: A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilitiescited by 8
  17. 2025Mind the Third Eye! Benchmarking Privacy Awareness in MLLM-powered Smartphone Agentscited by 8
  18. 2025MAS-Bench: A Unified Benchmark for Shortcut-Augmented Hybrid Mobile GUI Agentscited by 8
  19. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  20. 2025TAI3: Testing Agent Integrity in Interpreting User Intentcited by 7
  21. 2025DualTAP: A Dual-Task Adversarial Protector for Mobile MLLM Agentscited by 7
  22. 2026AmbiBench: Benchmarking Mobile GUI Agents Beyond One-Shot Instructions in the Wildcited by 7
  23. 2024MageBench: Bridging Large Multimodal Models to Agentscited by 6
  24. 2025EcoAgent: An Efficient Device-Cloud Collaborative Multi-Agent Framework for Mobile Automationcited by 6
  25. 2025TextAtari: 100K Frames Game Playing with Language Agentscited by 6
  26. 2026PhoneHarness: Harnessing Phone-Use Agents through Mixed GUI, CLI, and Tool Actionscited by 6
  27. 2025PerPilot: Personalizing VLM-based Mobile Agents via Memory and Explorationcited by 5
  28. 2025MPR-GUI: Benchmarking and Enhancing Multilingual Perception and Reasoning in GUI Agentscited by 5
  29. 2026VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnosticscited by 5
  30. 2025Towards Pervasive Distributed Agentic Generative AI -- A State of The Artcited by 4
  31. 2026MobileBench-OL: A Comprehensive Chinese Benchmark for Evaluating Mobile GUI Agents in Real-World Environmentcited by 4
  32. 2026PhoneBuddy: Training Open Models for Agentic Phone Usecited by 4
  33. 2025FedMABench: Benchmarking Mobile Agents on Decentralized Heterogeneous User Datacited by 3
  34. 2025MobiBench: Multi-Branch, Modular Benchmark for Mobile GUI Agentscited by 3
  35. 2025AndroidLens: Long-latency Evaluation with Nested Sub-targets for Android GUI Agentscited by 2
  36. 2026Do Multi-Agents Dream of Electric Screens? Achieving Perfect Accuracy on AndroidWorld Through Task Decompositioncited by 2
  37. 2026Jev-Mobile: Jev as an Executor for Mobile GUI Agentscited by 2
  38. 2024Probing the Capacity of Language Model Agents to Operationalize Disparate Experiential Context Despite Distractioncited by 1

50 most cited of 63 All 63 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agents
  2. 2026Benchmarking LLM Judges for Mobile Agent Evaluation
  3. 2025MVISU-Bench: Benchmarking Mobile Agents for Real-World Tasks by Multi-App, Vague, Interactive, Single-App and Unethical Instructions
  4. 2025MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments
  5. 2026SWE-Bench Mobile: Can Large Language Model Agents Develop Industry-Level Mobile Applications?
  6. 2025MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents
  7. 2026MobilePA-Bench: Benchmarking Mobile Planner Agents on Complex Real-World Tasks
  8. 2026Benchmarking General Mobile Assistants in Challenging Real-World Scenarios