mlsanityData to 7 Oct 2026

AppAgent v2: Advanced Agent for Flexible Mobile Interactions

arXiv 2408.11824 ↗First posted 5 Aug 2024cs.HC cs.AIcited by 67

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

With the advancement of Multimodal Large Language Models (MLLM), LLM-driven visual agents are increasingly impacting software interfaces, particularly those with graphical user interfaces. This work introduces a novel LLM-based multimodal agent framework for mobile devices. This framework, capable of navigating mobile devices, emulates human-like interactions. Our agent constructs a flexible action space that enhances adaptability across various applications including parser, text and vision descriptions. The agent operates through two main phases: exploration and deployment. During the exploration phase, functionalities of user interface elements are documented either through agent-driven or manual explorations into a customized structured knowledge base. In the deployment phase, RAG technology enables efficient retrieval and update from this knowledge base, thereby empowering the agent to perform tasks effectively and accurately. This includes performing complex, multi-step operations across various applications, thereby demonstrating the framework's adaptability and precision in handling customized task workflows. Our experimental results across various benchmarks demonstrate the framework's superior performance, confirming its effectiveness in real-world scenarios. Our code will be open source soon.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 25 of its 40 references that are in the map. First posted on arXiv 5 Aug 2024.

Builds on

25 of 40 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception29 Jan
  2. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan

Other references · 23

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  3. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  4. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  5. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  6. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  7. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  8. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  9. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  10. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  11. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 245
  12. 2019Real-time Scene Text Detection with Differentiable Binarizationcited by 212
11 more
  1. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 188
  2. 2023AutoDroid: LLM-powered Task Automation in Androidcited by 178
  3. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agentscited by 123
  4. 2023Discuss Before Moving: Visual Language Navigation via Multi-expert Discussionscited by 108
  5. 2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learncited by 97
  6. 2023On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Drivingcited by 79
  7. 2023Octopus: Embodied Vision-Language Programmer from Environmental Feedbackcited by 73
  8. 2023MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perceptioncited by 67
  9. 2023ChatVideo: A Tracklet-centric Multimodal and Versatile Video Understanding Systemcited by 65
  10. 2023DroidBot-GPT: GPT-powered UI Automation for Androidcited by 47
  11. 2023StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Datacited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 43; 2026: 20.

67 papers.

Sort:

By area:

  1. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 183
  2. 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learningcited by 166
  3. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  4. 2025Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Taskscited by 128
  5. 2025Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learningcited by 79
  6. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  7. 2024Ferret-UI 2: Mastering Universal User Interface Understanding Across Platformscited by 51
  8. 2025AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuningcited by 44
  9. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  10. 2025Advancing Mobile GUI Agents: A Verifier-Driven Approach to Practical Deploymentcited by 38
  11. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  12. 2025A Comprehensive Survey of Agents for Computer Use: Foundations, Challenges, and Future Directionscited by 33
38 more
  1. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  2. 2025Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automationcited by 27
  3. 2025ManipLVM-R1: Reinforcement Learning for Reasoning in Embodied Manipulation with Large Vision-Language Modelscited by 26
  4. 2024Falcon-UI: Understanding GUI Before Following User Instructionscited by 22
  5. 2025GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behaviorcited by 22
  6. 2025MobileWorldBench: Towards Semantic World Modeling For Mobile Agentscited by 18
  7. 2025Mobile-Bench-v2: A More Realistic and Comprehensive Benchmark for VLM-based Mobile Agentscited by 16
  8. 2025Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Modelscited by 15
  9. 2025Uncertainty-Aware GUI Agent: Adaptive Perception through Component Recommendation and Human-in-the-Loop Refinementcited by 15
  10. 2025Chain-of-Memory: Enhancing GUI Agents for Cross-Application Navigationcited by 13
  11. 2025Agent-SAMA: State-Aware Mobile Assistantcited by 12
  12. 2025HALO: Hierarchical Autonomous Logic-Oriented Orchestration for Multi-Agent LLM Systemscited by 11
  13. 2025Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling Systemcited by 11
  14. 2025From Assistants to Adversaries: Exploring the Security Risks of Mobile LLM Agentscited by 10
  15. 2025MobileIPL: Enhancing Mobile Agents Thinking Process via Iterative Preference Learningcited by 9
  16. 2025GEM: Gaussian Embedding Modeling for Out-of-Distribution Detection in GUI Agentscited by 8
  17. 2025Mind the Third Eye! Benchmarking Privacy Awareness in MLLM-powered Smartphone Agentscited by 8
  18. 2026MAGNET: Towards Adaptive GUI Agents with Memory-Driven Knowledge Evolutioncited by 8
  19. 2026AmbiBench: Benchmarking Mobile GUI Agents Beyond One-Shot Instructions in the Wildcited by 7
  20. 2024MageBench: Bridging Large Multimodal Models to Agentscited by 6
  21. 2025CRAFT-GUI: Curriculum-Reinforced Agent For GUI Taskscited by 6
  22. 2025PG-Agent: An Agent Powered by Page Graphcited by 6
  23. 2025ColorAgent: Building A Robust, Personalized, and Interactive OS Agentcited by 6
  24. 2026ColorBrowserAgent: Complex Long-Horizon Browser Agent with Adaptive Knowledge Evolutioncited by 6
  25. 2026Me-Agent: A Personalized Mobile Agent with Two-Level User Habit Learning for Enhanced Interactioncited by 6
  26. 2026Generalization in Online Reinforcement Learning for Mobile Agentscited by 6
  27. 2025AVA: Attentive VLM Agent for Mastering StarCraft IIcited by 5
  28. 2025Mobile-Agent-RAG: Driving Smart Multi-Agent Coordination with Contextual Knowledge Empowerment for Long-Horizon Mobile Automationcited by 5
  29. 2026ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence on Mobile Devicescited by 5
  30. 2026What Memory Do GUI Agents Really Need? From Passive Records to Active Task-Driving Statescited by 5
  31. 2025DeskVision: Large Scale Desktop Region Captioning for Advanced GUI Agentscited by 4
  32. 2025CORE: Reducing UI Exposure in Mobile Agents via Collaboration Between Cloud and Local LLMscited by 4
  33. 2025GUI Exploration Lab: Enhancing Screen Navigation in Agents via Multi-Turn Reinforcement Learningcited by 4
  34. 2025Agent-ScanKit: Unraveling Memory and Reasoning of Multimodal Agents via Sensitivity Perturbationscited by 3
  35. 2025RISK: A Framework for GUI Agents in E-commerce Risk Managementcited by 2
  36. 2026IntentCUA: Learning Intent-level Representations for Skill Abstraction and Multi-Agent Planning in Computer-Use Agentscited by 2
  37. 2026AgentRAE: Remote Action Execution through Notification-based Visual Backdoors against Screenshots-based Mobile GUI Agentscited by 2
  38. 2026Mobile GUI Agent Privacy Personalization with Trajectory Induced Preference Optimizationcited by 2

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023AppAgent: Multimodal Agents as Smartphone Users
  2. 2024Foundations and Recent Trends in Multimodal Mobile Agents: A Survey
  3. 2024MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents
  4. 2025AppCopilot: Toward General, Accurate, Long-Horizon, and Efficient Mobile Agent
  5. 2024AXIS: Efficient Human-Agent-Computer Interaction with API-First LLM-Based Agents
  6. 2025API Agents vs. GUI Agents: Divergence and Convergence
  7. 2025MobiAgent: A Systematic Framework for Customizable Mobile Agents
  8. 2024Understanding the Weakness of Large Language Model Agents within a Complex Android Environment