mlsanityData to 7 Oct 2026

Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception

arXiv 2401.16158 ↗First posted 29 Jan 2024cs.CL cs.CVcited by 283

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

Mobile device agent based on Multimodal Large Language Models (MLLM) is becoming a popular application. In this paper, we introduce Mobile-Agent, an autonomous multi-modal mobile device agent. Mobile-Agent first leverages visual perception tools to accurately identify and locate both the visual and textual elements within the app's front-end interface. Based on the perceived vision context, it then autonomously plans and decomposes the complex operation task, and navigates the mobile Apps through operations step by step. Different from previous solutions that rely on XML files of Apps or mobile system metadata, Mobile-Agent allows for greater adaptability across diverse mobile operating environments in a vision-centric way, thereby eliminating the necessity for system-specific customizations. To assess the performance of Mobile-Agent, we introduced Mobile-Eval, a benchmark for evaluating mobile device operations. Based on Mobile-Eval, we conducted a comprehensive evaluation of Mobile-Agent. The experimental results indicate that Mobile-Agent achieved remarkable accuracy and completion rates. Even with challenging instructions, such as multi-app operations, Mobile-Agent can still complete the requirements. Code and model will be open-sourced at https://github.com/X-PLUG/MobileAgent.

Key facts. Cited by 283 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 23 of its 25 references that are in the map. First posted on arXiv 29 Jan 2024.

Builds on

23 of 25 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023AppAgent: Multimodal Agents as Smartphone Users21 Dec

Other references · 22

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  7. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  8. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  9. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  10. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  11. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  12. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
10 more
  1. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  2. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  3. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  4. 2023Auto-GPT for Online Decision Making: Benchmarks and Additional Opinionscited by 250
  5. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 245
  6. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 188
  7. 2023InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Languagecited by 96
  8. 2024Small LLMs Are Weak Tool Learners: A Multi-LLM Agentcited by 92
  9. 2023ControlLLM: Augment Language Models with Tools by Searching on Graphscited by 59
  10. 2023ModelScope-Agent: Building Your Customizable Agent System with Open-source Large Language Modelscited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 63; 2025: 126; 2026: 94.

283 papers.

Sort:

By area:

  1. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  2. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  3. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  4. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 201
  5. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  6. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
  7. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaborationcited by 187
  8. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 173
  9. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  10. 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learningcited by 166
  11. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  12. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesiscited by 155
38 more
  1. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 149
  2. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 147
  3. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
  4. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 136
  5. 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concretecited by 134
  6. 2025Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Taskscited by 128
  7. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 127
  8. 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 114
  9. 2024GUICourse: From General Vision Language Models to Versatile GUI Agentscited by 112
  10. 2024GUI Agents with Foundation Models: A Comprehensive Surveycited by 110
  11. 2024MMedAgent: Learning to Use Medical Tools with Multi-modal Agentcited by 108
  12. 2024VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agentscited by 101
  13. 2024GUI Agents: A Surveycited by 101
  14. 2024Large Multimodal Agents: A Surveycited by 99
  15. 2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 90
  16. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 89
  17. 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agentscited by 88
  18. 2024SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluationcited by 79
  19. 2025Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learningcited by 79
  20. 2024o1-Coder: an o1 Replication for Codingcited by 77
  21. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  22. 2026Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agentscited by 72
  23. 2024AppAgent v2: Advanced Agent for Flexible Mobile Interactionscited by 67
  24. 2024MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agentscited by 63
  25. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  26. 2025GUI-G2: Gaussian Reward Modeling for GUI Groundingcited by 60
  27. 2025VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agentscited by 60
  28. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 58
  29. 2024MMInA: Benchmarking Multihop Multimodal Internet Agentscited by 55
  30. 2024Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?cited by 55
  31. 2024Ferret-UI 2: Mastering Universal User Interface Understanding Across Platformscited by 51
  32. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  33. 2024Octopus v2: On-device language model for super agentcited by 45
  34. 2025AppAgentX: Evolving GUI Agents as Proficient Smartphone Userscited by 44
  35. 2025ZeroGUI: Automating Online GUI Learning at Zero Human Costcited by 44
  36. 2025AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuningcited by 44
  37. 2025Fully Autonomous AI Agents Should Not be Developedcited by 43
  38. 2025Agentic Web: Weaving the Next Web with AI Agentscited by 43

50 most cited of 283 All 283 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MobiAgent: A Systematic Framework for Customizable Mobile Agents
  2. 2025AppCopilot: Toward General, Accurate, Long-Horizon, and Efficient Mobile Agent
  3. 2026(A)I Sees What You Don't: Exploiting New Attack Surfaces in Third-Party Mobile Agents
  4. 2025MobileUse: A GUI Agent with Hierarchical Reflection for Autonomous Mobile Operation
  5. 2026PalmClaw: A Native On-Device Agent Framework for Mobile Phones
  6. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices
  7. 2024MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understanding
  8. 2025MobileWorldBench: Towards Semantic World Modeling For Mobile Agents