mlsanityData to 7 Oct 2026

OmniParser for Pure Vision Based GUI Agent

arXiv 2408.00203 ↗First posted 1 Aug 2024cs.CV cs.AI cs.CLcited by 193

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

The recent success of large vision language models shows great potential in driving the agent system operating on user interfaces. However, we argue that the power multimodal models like GPT-4V as a general agent on multiple operating systems across different applications is largely underestimated due to the lack of a robust screen parsing technique capable of: 1) reliably identifying interactable icons within the user interface, and 2) understanding the semantics of various elements in a screenshot and accurately associate the intended action with the corresponding region on the screen. To fill these gaps, we introduce textscOmniParser, a comprehensive method for parsing user interface screenshots into structured elements, which significantly enhances the ability of GPT-4V to generate actions that can be accurately grounded in the corresponding regions of the interface. We first curated an interactable icon detection dataset using popular webpages and an icon description dataset. These datasets were utilized to fine-tune specialized models: a detection model to parse interactable regions on the screen and a caption model to extract the functional semantics of the detected elements. textscOmniParser significantly improves GPT-4V's performance on ScreenSpot benchmark. And on Mind2Web and AITW benchmark, textscOmniParser with screenshot only input outperforms the GPT-4V baselines requiring additional information outside of screenshot.

Key facts. Cited by 193 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 14 of its 22 references that are in the map. First posted on arXiv 1 Aug 2024.

Builds on

14 of 22 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec
  2. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation13 Nov

Other references · 12

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  3. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  4. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
  5. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
  6. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  7. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 119
  8. 2020Widget Captioning: Generating Natural Language Description for Mobile User Interface Elementscited by 94
  9. 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 93
  10. 2023From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfacescited by 67
  11. 2020ActionBert: Leveraging User Actions for Semantic Understanding of User Interfacescited by 43
  12. 2022Towards Better Semantic Understanding of Mobile Interfacescited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 100; 2026: 79.

193 papers.

Sort:

By area:

  1. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  2. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  3. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 370
  4. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
  5. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  6. 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scalecited by 212
  7. 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learningcited by 166
  8. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  9. 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agentscited by 156
  10. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
  11. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 132
  12. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
38 more
  1. 2024AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorialscited by 123
  2. 2025Agentic Large Language Models, a surveycited by 105
  3. 2025GTA1: GUI Test-time Scaling Agentcited by 102
  4. 2024GUI Agents: A Surveycited by 101
  5. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 89
  6. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
  7. 2024DINO-X: A Unified Vision Model for Open-World Object Detection and Understandingcited by 85
  8. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  9. 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agentscited by 73
  10. 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 68
  11. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  12. 2025GUI-G2: Gaussian Reward Modeling for GUI Groundingcited by 60
  13. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 58
  14. 2025AppAgentX: Evolving GUI Agents as Proficient Smartphone Userscited by 44
  15. 2025AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuningcited by 44
  16. 2025UFO2: The Desktop AgentOScited by 38
  17. 2025OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agentscited by 37
  18. 2025Fara-7B: An Efficient Agentic Model for Computer Usecited by 34
  19. 2025A Comprehensive Survey of Agents for Computer Use: Foundations, Challenges, and Future Directionscited by 33
  20. 2025CoAct-1: Computer-using Multi-Agent System with Coding Actionscited by 33
  21. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  22. 2025macOSWorld: A Multilingual Interactive Benchmark for GUI Agentscited by 31
  23. 2025DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoningcited by 29
  24. 2025RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Codecited by 28
  25. 2025API Agents vs. GUI Agents: Divergence and Convergencecited by 28
  26. 2025FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflowcited by 25
  27. 2025AgentSentinel: An End-to-End and Real-Time Security Defense Framework for Computer-Use Agentscited by 25
  28. 2025Phi-Ground Tech Report: Advancing Perception in GUI Groundingcited by 24
  29. 2025Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curationcited by 24
  30. 2024Falcon-UI: Understanding GUI Before Following User Instructionscited by 22
  31. 2025GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behaviorcited by 22
  32. 2025UI-E2I-Synth: Advancing GUI Grounding with Large-Scale Instruction Synthesiscited by 20
  33. 2025UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoningcited by 20
  34. 2025MagicGUI: A Foundational Mobile GUI Agent with Scalable Data Pipeline and Reinforcement Fine-tuningcited by 19
  35. 2025GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Explorationcited by 18
  36. 2025Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skillscited by 18
  37. 2025Morae: Proactively Pausing UI Agents for User Choicescited by 18
  38. 2024MolParser: End-to-end Visual Recognition of Molecule Structures in the Wildcited by 17

50 most cited of 193 All 193 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environments
  2. 2024Ponder & Press: Advancing Visual GUI Agent towards General Computer Control
  3. 2024GUICourse: From General Vision Language Models to Versatile GUI Agents
  4. 2024TinyClick: Single-Turn Agent for Empowering GUI Automation
  5. 2025SpiritSight Agent: Advanced GUI Agent with One Look
  6. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understanding
  7. 2025MEGA-GUI: Multi-stage Enhanced Grounding Agents for GUI Elements
  8. 2025TRISHUL: Towards Region Identification and Screen Hierarchy Understanding for Large VLM based GUI Agents