mlsanityData to 7 Oct 2026

Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents

arXiv 2410.05243 ↗First posted 7 Oct 2024cs.AI cs.CL cs.CVcited by 373

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Multimodal large language models (MLLMs) are transforming the capabilities of graphical user interface (GUI) agents, facilitating their transition from controlled simulations to complex, real-world applications across various platforms. However, the effectiveness of these agents hinges on the robustness of their grounding capability. Current GUI agents predominantly utilize text-based representations such as HTML or accessibility trees, which, despite their utility, often introduce noise, incompleteness, and increased computational overhead. In this paper, we advocate a human-like embodiment for GUI agents that perceive the environment entirely visually and directly perform pixel-level operations on the GUI. The key is visual grounding models that can accurately map diverse referring expressions of GUI elements to their coordinates on the GUI across different platforms. We show that a simple recipe, which includes web-based synthetic data and slight adaptation of the LLaVA architecture, is surprisingly effective for training such visual grounding models. We collect the largest dataset for GUI visual grounding so far, containing 10M GUI elements and their referring expressions over 1.3M screenshots, and use it to train UGround, a strong universal visual grounding model for GUI agents. Empirical results on six benchmarks spanning three categories (grounding, offline agent, and online agent) show that 1) UGround substantially outperforms existing visual grounding models for GUI agents, by up to 20% absolute, and 2) agents with UGround outperform state-of-the-art agents, despite the fact that existing agents use additional text-based input while ours only uses visual perception. These results provide strong support for the feasibility and promises of GUI agents that navigate the digital world as humans do.

Key facts. Cited by 373 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 56 of its 70 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

56 of 70 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024OmniParser for Pure Vision Based GUI Agent1 Aug
  2. 2024GUICourse: From General Vision Language Models to Versatile GUI Agents17 Jun
  3. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs8 Apr
  4. 2024ScreenAgent: A Vision Language Model-driven Computer Control Agent9 Feb
  5. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models25 Jan
  6. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec
  7. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation13 Nov
  8. 2022Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus29 Sep

Other references · 48

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  8. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  9. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  10. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  11. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  12. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
36 more
  1. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  2. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  3. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  4. 2016Modeling Context in Referring Expressionscited by 1,054
  5. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  6. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  7. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  8. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  9. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  10. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  11. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  12. 2023Language Models can Solve Computer Taskscited by 371
  13. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  14. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  15. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
  16. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvementcited by 203
  17. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 193
  18. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  19. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
  20. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 156
  21. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agentscited by 156
  22. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  23. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 147
  24. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 130
  25. 2024WebCanvas: Benchmarking Web Agents in Online Environmentscited by 119
  26. 2024Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Modelscited by 118
  27. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 114
  28. 2014Deep Fragment Embeddings for Bidirectional Image Sentence Mappingcited by 99
  29. 2020Widget Captioning: Generating Natural Language Description for Mobile User Interface Elementscited by 94
  30. 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 93
  31. 2022A data-driven approach for learning to control computerscited by 87
  32. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  33. 2023From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfacescited by 67
  34. 2024Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?cited by 55
  35. 2023Can We Edit Multimodal Large Language Models?cited by 44
  36. 2023Lexi: Self-Supervised Learning of the UI Languagecited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 198; 2026: 157.

373 papers.

Sort:

By area:

  1. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  2. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  3. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  4. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 370
  5. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
  6. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 246
  7. 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agentscited by 240
  8. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  9. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  10. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 183
  11. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 174
  12. 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learningcited by 166
38 more
  1. 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 165
  2. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  3. 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agentscited by 156
  4. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesiscited by 155
  5. 2024The BrowserGym Ecosystem for Web Agent Researchcited by 152
  6. 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 151
  7. 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agentscited by 144
  8. 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasonerscited by 140
  9. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
  10. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 132
  11. 2025GTA1: GUI Test-time Scaling Agentcited by 102
  12. 2024GUI Agents: A Surveycited by 101
  13. 2024AutoGLM: Autonomous Foundation Agents for GUIscited by 96
  14. 2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 90
  15. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 89
  16. 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agentscited by 88
  17. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
  18. 2025AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detectioncited by 82
  19. 2025Grounded Reinforcement Learning for Visual Reasoningcited by 80
  20. 2025Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learningcited by 79
  21. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  22. 2025ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replaycited by 75
  23. 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agentscited by 73
  24. 2026Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agentscited by 72
  25. 2025Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agentscited by 68
  26. 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 68
  27. 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 68
  28. 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflectioncited by 67
  29. 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 64
  30. 2024The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Usecited by 62
  31. 2025GUI-G2: Gaussian Reward Modeling for GUI Groundingcited by 60
  32. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 58
  33. 2025Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generationcited by 54
  34. 2025MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environmentcited by 51
  35. 2026EvoCUA: Evolving Computer Use Agents via Learning from Scalable Synthetic Experiencecited by 51
  36. 2025ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflowscited by 47
  37. 2025ZeroGUI: Automating Online GUI Learning at Zero Human Costcited by 44
  38. 2025AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuningcited by 44

50 most cited of 373 All 373 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Ponder & Press: Advancing Visual GUI Agent towards General Computer Control
  2. 2025DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoning
  3. 2025Chain-of-Ground: Improving GUI Grounding via Iterative Reasoning and Reference Feedback
  4. 2024GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding
  5. 2024EDGE: Enhanced Grounded GUI Understanding with Enriched Multi-Granularity Synthetic Data
  6. 2026Trifuse: Enhancing Attention-Based GUI Grounding via Multimodal Fusion
  7. 2025Learning GUI Grounding with Spatial Reasoning from Visual Feedback
  8. 2026GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents