mlsanityData to 7 Oct 2026

GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation

arXiv 2311.07562 ↗First posted 13 Nov 2023cs.CV cs.AIcited by 137

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

We present MM-Navigator, a GPT-4V-based agent for the smartphone graphical user interface (GUI) navigation task. MM-Navigator can interact with a smartphone screen as human users, and determine subsequent actions to fulfill given instructions. Our findings demonstrate that large multimodal models (LMMs), specifically GPT-4V, excel in zero-shot GUI navigation through its advanced screen interpretation, action reasoning, and precise action localization capabilities. We first benchmark MM-Navigator on our collected iOS screen dataset. According to human assessments, the system exhibited a 91% accuracy rate in generating reasonable action descriptions and a 75% accuracy rate in executing the correct actions for single-step instructions on iOS. Additionally, we evaluate the model on a subset of an Android screen navigation dataset, where the model outperforms previous GUI navigators in a zero-shot fashion. Our benchmark and detailed analyses aim to lay a robust groundwork for future research into the GUI navigation task. The project page is at https://github.com/zzxslp/MM-Navigator.

Key facts. Cited by 137 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 58 references that are in the map. First posted on arXiv 13 Nov 2023.

Builds on

48 of 58 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 47

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2023Segment Anythingcited by 8,158
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  11. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  12. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
35 more
  1. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  2. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  3. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  4. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  5. 2016Modeling Context in Referring Expressionscited by 1,054
  6. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  7. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  8. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  9. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  10. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  11. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  12. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  13. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  14. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  15. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  16. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  17. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  18. 2018Model compression via distillation and quantizationcited by 287
  19. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
  20. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  21. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  22. 2019A Fast and Accurate One-Stage Approach to Visual Groundingcited by 194
  23. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  24. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
  25. 2023AutoDroid: LLM-powered Task Automation in Androidcited by 178
  26. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  27. 2023GPT-4V(ision) as a Generalist Evaluator for Vision-Language Taskscited by 118
  28. 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 93
  29. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 82
  30. 2022META-GUI: Towards Multi-modal Conversational Agents on Mobile GUIcited by 81
  31. 2021Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixelscited by 57
  32. 2023Idea2Img: Iterative Self-Refinement with GPT-4V(ision) for Automatic Image Design and Generationcited by 38
  33. 2021Mobile App Tasks with Iterative Feedback (MoTIF): Addressing Task Feasibility in Interactive Visual Environmentscited by 28
  34. 2022Towards Better Semantic Understanding of Mobile Interfacescited by 24
  35. 2023Learning to Plan with Natural Languagecited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 62; 2025: 56; 2026: 16.

137 papers.

Sort:

By area:

  1. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  2. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  3. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  4. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  5. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  6. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 424
  7. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  8. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  9. 2024TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policiescited by 298
  10. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 260
  11. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  12. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
38 more
  1. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  2. 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scalecited by 212
  3. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 196
  4. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  5. 2024OmniParser for Pure Vision Based GUI Agentcited by 193
  6. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
  7. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
  8. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaborationcited by 187
  9. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  10. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  11. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agentscited by 156
  12. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 147
  13. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
  14. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 136
  15. 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 114
  16. 2024GUICourse: From General Vision Language Models to Versatile GUI Agentscited by 112
  17. 2024GUI Agents with Foundation Models: A Comprehensive Surveycited by 110
  18. 2024AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agentscited by 105
  19. 2024Large Multimodal Agents: A Surveycited by 99
  20. 2024CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automationcited by 70
  21. 2025Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agentscited by 68
  22. 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 68
  23. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  24. 2024Understanding the Weakness of Large Language Model Agents within a Complex Android Environmentcited by 60
  25. 2024MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilitiescited by 57
  26. 2024Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental Distractionscited by 55
  27. 2025The Obvious Invisible Threat: LLM-Powered GUI Agents' Vulnerability to Fine-Print Injectionscited by 55
  28. 2024Benchmarking Mobile Device Control Agents across Diverse Configurationscited by 54
  29. 2024MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understandingcited by 46
  30. 2023ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automationcited by 45
  31. 2024VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Taskscited by 45
  32. 2025Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agentscited by 43
  33. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMscited by 38
  34. 2024End-to-End Navigation with Vision Language Models: Transforming Spatial Reasoning into Question-Answeringcited by 37
  35. 2024LlamaTouch: A Faithful and Scalable Testbed for Mobile UI Task Automationcited by 36
  36. 2024VideoGUI: A Benchmark for GUI Automation from Instructional Videoscited by 35
  37. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  38. 2025A Comprehensive Survey of Agents for Computer Use: Foundations, Challenges, and Future Directionscited by 33

50 most cited of 137 All 137 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024MobileViews: A Million-scale and Diverse Mobile GUI Dataset
  2. 2026OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environments
  3. 2026From Tapping to Hopping: Augmenting Mobile GUI Agents with App-Native Deeplinks
  4. 2025MagicGUI: A Foundational Mobile GUI Agent with Scalable Data Pipeline and Reinforcement Fine-tuning
  5. 2026GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agents
  6. 2026AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agents
  7. 2023Mobile-Env: Building Qualified Evaluation Benchmarks for LLM-GUI Interaction
  8. 2026POINTS-GUI-G: GUI-Grounding Journey