mlsanityData to 14 Oct 2026

AppAgent: Multimodal Agents as Smartphone Users

arXiv 2312.13771 ↗First posted 21 Dec 2023cs.CVcited by 415

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

Recent advancements in large language models (LLMs) have led to the creation of intelligent agents capable of performing complex tasks. This paper introduces a novel LLM-based multimodal agent framework designed to operate smartphone applications. Our framework enables the agent to operate smartphone applications through a simplified action space, mimicking human-like interactions such as tapping and swiping. This novel approach bypasses the need for system back-end access, thereby broadening its applicability across diverse apps. Central to our agent's functionality is its innovative learning method. The agent learns to navigate and use new apps either through autonomous exploration or by observing human demonstrations. This process generates a knowledge base that the agent refers to for executing complex tasks across different applications. To demonstrate the practicality of our agent, we conducted extensive testing over 50 tasks in 10 different applications, including social media, email, maps, shopping, and sophisticated image editing tools. The results affirm our agent's proficiency in handling a diverse array of high-level tasks.

Key facts. Cited by 415 papers in the map (data through 14 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 27 of its 53 references that are in the map. First posted on arXiv 21 Dec 2023.

Builds on

27 of 53 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec
  2. 2023You Only Look at Screens: Multimodal Chain-of-Action Agents20 Sep

Other references · 25

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  8. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  9. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  10. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  11. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  12. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
13 more
  1. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  2. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 574
  3. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353
  4. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 319
  5. 2023Auto-GPT for Online Decision Making: Benchmarks and Additional Opinionscited by 255
  6. 2023Exploring Large Language Models for Communication Games: An Empirical Study on Werewolfcited by 241
  7. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 171
  8. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 160
  9. 2023OpenAgents: An Open Platform for Language Agents in the Wildcited by 137
  10. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  11. 2023Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planningcited by 60
  12. 2023StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Datacited by 33
  13. 2023Never-ending Learning of User Interfacescited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 107; 2025: 155; 2026: 153.

415 papers.

Sort:

By area:

  1. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 610
  2. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
  3. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 438
  4. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 339
  5. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 293
  6. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 288
  7. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 278
  8. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 256
  9. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 238
  10. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 214
  11. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 208
  12. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvementcited by 205
38 more
  1. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 203
  2. 2025Plan-and-Act: Improving Planning of Agents for Long-Horizon Taskscited by 195
  3. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 194
  4. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 193
  5. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 190
  6. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaborationcited by 189
  7. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 174
  8. 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learningcited by 168
  9. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesiscited by 163
  10. 2024Large Language Model-Brained GUI Agents: A Surveycited by 162
  11. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agentscited by 159
  12. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 153
  13. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 148
  14. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 140
  15. 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasonerscited by 140
  16. 2024A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalistcited by 137
  17. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 137
  18. 2024GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editingcited by 133
  19. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 131
  20. 2026SoK: Agentic Skills -- Beyond Tool Use in LLM Agentscited by 130
  21. 2024Dissecting Adversarial Robustness of Multimodal LM Agentscited by 120
  22. 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 117
  23. 2024GUICourse: From General Vision Language Models to Versatile GUI Agentscited by 114
  24. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 113
  25. 2024GUI Agents with Foundation Models: A Comprehensive Surveycited by 111
  26. 2024Large Multimodal Agents: A Surveycited by 105
  27. 2024VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agentscited by 103
  28. 2024GUI Agents: A Surveycited by 103
  29. 2024ScreenAgent: A Vision Language Model-driven Computer Control Agentcited by 102
  30. 2024AutoGLM: Autonomous Foundation Agents for GUIscited by 100
  31. 2025Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in Realistic Environmentscited by 95
  32. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 95
  33. 2024Agent-Pro: Learning to Evolve via Policy-Level Reflection and Optimizationcited by 94
  34. 2024Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Taskscited by 84
  35. 2024SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluationcited by 81
  36. 2025Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learningcited by 81
  37. 2026Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agentscited by 79
  38. 2024Beyond Browsing: API-Based Web Agentscited by 78

50 most cited of 415 All 415 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024AppAgent v2: Advanced Agent for Flexible Mobile Interactions
  2. 2024Foundations and Recent Trends in Multimodal Mobile Agents: A Survey
  3. 2025AppCopilot: Toward General, Accurate, Long-Horizon, and Efficient Mobile Agent
  4. 2026PalmClaw: A Native On-Device Agent Framework for Mobile Phones
  5. 2025Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Tasks
  6. 2025Agent-Initiated Interaction in Phone UI Automation
  7. 2024Understanding the Weakness of Large Language Model Agents within a Complex Android Environment
  8. 2025API Agents vs. GUI Agents: Divergence and Convergence