mlsanityData to 7 Oct 2026

ShowUI: One Vision-Language-Action Model for GUI Visual Agent

arXiv 2411.17465 ↗First posted 26 Nov 2024cs.CV cs.AI cs.CLcited by 226

Not on the map. Why?

Cited by 144 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Building Graphical User Interface (GUI) assistants holds significant promise for enhancing human workflow productivity. While most agents are language-based, relying on closed-source API with text-rich meta-information (e.g., HTML or accessibility tree), they show limitations in perceiving UI visuals as humans do, highlighting the need for GUI visual agents. In this work, we develop a vision-language-action model in digital world, namely ShowUI, which features the following innovations: (i) UI-Guided Visual Token Selection to reduce computational costs by formulating screenshots as an UI connected graph, adaptively identifying their redundant relationship and serve as the criteria for token selection during self-attention blocks; (ii) Interleaved Vision-Language-Action Streaming that flexibly unifies diverse needs within GUI tasks, enabling effective management of visual-action history in navigation or pairing multi-turn query-action sequences per screenshot to enhance training efficiency; (iii) Small-scale High-quality GUI Instruction-following Datasets by careful data curation and employing a resampling strategy to address significant data type imbalances. With above components, ShowUI, a lightweight 2B model using 256K data, achieves a strong 75.1% accuracy in zero-shot screenshot grounding. Its UI-guided token selection further reduces 33% of redundant visual tokens during training and speeds up the performance by 1.4x. Navigation experiments across web Mind2Web, mobile AITW, and online MiniWob environments further underscore the effectiveness and potential of our model in advancing GUI visual agents. The models are available at https://github.com/showlab/ShowUI.

Key facts. Cited by 226 papers in the map (data through 7 Oct 2026). Builds on 47 of its 52 references that are in the map. First posted on arXiv 26 Nov 2024.

Builds on

47 of 52 references.

References · 47

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  4. 2023Qwen Technical Reportcited by 3,273
  5. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  6. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  7. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  8. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  9. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  10. 2022Token Merging: Your ViT But Fastercited by 995
  11. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  12. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
35 more
  1. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  2. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  3. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  4. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  5. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  6. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
  7. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  8. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  9. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  10. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  11. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 320
  12. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  13. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 245
  14. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  15. 2024OmniParser for Pure Vision Based GUI Agentcited by 193
  16. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
  17. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  18. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
  19. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  20. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  21. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 156
  22. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  23. 2024GUICourse: From General Vision Language Models to Versatile GUI Agentscited by 112
  24. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 110
  25. 2024xLAM: A Family of Large Action Models to Empower AI Agent Systemscited by 108
  26. 2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learncited by 97
  27. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 95
  28. 2022A data-driven approach for learning to control computerscited by 87
  29. 2024CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Drivingcited by 74
  30. 2023From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfacescited by 67
  31. 2024VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computationcited by 55
  32. 2024Ferret-UI 2: Mastering Universal User Interface Understanding Across Platformscited by 51
  33. 2023WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semanticscited by 45
  34. 2024VideoGUI: A Benchmark for GUI Automation from Instructional Videoscited by 35
  35. 2024A3VLM: Actionable Articulation-Aware Vision Language Modelcited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 107; 2026: 117.

226 papers.

Sort:

By area:

  1. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  2. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  3. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 246
  4. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 203
  5. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  6. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 183
  7. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  8. 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 151
  9. 2024GUI Agents: A Surveycited by 101
  10. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 89
  11. 2025Grounded Reinforcement Learning for Visual Reasoningcited by 80
  12. 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agentscited by 73
38 more
  1. 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 68
  2. 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 68
  3. 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 64
  4. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  5. 2025GUI-G2: Gaussian Reward Modeling for GUI Groundingcited by 60
  6. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 58
  7. 2025Paper2Poster: Towards Multimodal Poster Automation from Scientific Paperscited by 56
  8. 2025FineVision: Open Data Is All You Needcited by 51
  9. 2026EvoCUA: Evolving Computer Use Agents via Learning from Scalable Synthetic Experiencecited by 51
  10. 2025ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflowscited by 47
  11. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 46
  12. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  13. 2025ZeroGUI: Automating Online GUI Learning at Zero Human Costcited by 44
  14. 2025AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuningcited by 44
  15. 2025TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agentscited by 40
  16. 2026Code as Agent Harnesscited by 40
  17. 2025Vision Language Action Models in Robotic Manipulation: A Systematic Reviewcited by 38
  18. 2025RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agentscited by 34
  19. 2025UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Groundingcited by 32
  20. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  21. 2025macOSWorld: A Multilingual Interactive Benchmark for GUI Agentscited by 31
  22. 2025Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Modelscited by 30
  23. 2025One RL to See Them All: Visual Triple Unified Reinforcement Learningcited by 29
  24. 2025DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoningcited by 29
  25. 2026OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Taskscited by 29
  26. 2026Code2World: A GUI World Model via Renderable Code Generationcited by 26
  27. 2025BacktrackAgent: Enhancing GUI Agent with Error Detection and Backtracking Mechanismcited by 25
  28. 2025Why Vision Language Models Struggle with Visual Arithmetic? Towards Enhanced Chart and Geometry Understandingcited by 23
  29. 2025Think Twice, Click Once: Enhancing GUI Grounding via Fast and Slow Systemscited by 23
  30. 2025Code2Video: A Code-centric Paradigm for Educational Video Generationcited by 23
  31. 2025Paper2Video: Automatic Video Generation from Scientific Paperscited by 23
  32. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
  33. 2025UItron: Foundational GUI Agent with Advanced Perception and Planningcited by 22
  34. 2025PresentAgent: Multimodal Agent for Presentation Video Generationcited by 20
  35. 2025UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoningcited by 20
  36. 2026Workspace-Bench 1.0: Benchmarking AI Agents on Workspace Tasks with Large-Scale File Dependenciescited by 20
  37. 2025WorldGUI: An Interactive Benchmark for Desktop GUI Automation from Any Starting Pointcited by 19
  38. 2025VEM: Environment-Free Exploration for Training GUI Agent with Value Environment Modelcited by 19

50 most cited of 226 All 226 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025ScreenLLM: Stateful Screen Schema for Efficient Action Understanding and Prediction
  2. 2026Imagine to Act: High-Fidelity Data Synthesis via Image Editing World Model for Scalable GUI Agent Training
  3. 2026AutoGUI-v2: A Comprehensive Multi-Modal GUI Functionality Understanding Benchmark
  4. 2025Breaking the Data Barrier -- Building GUI Agents Through Task Generalization
  5. 2024Falcon-UI: Understanding GUI Before Following User Instructions
  6. 2024Ponder & Press: Advancing Visual GUI Agent towards General Computer Control
  7. 2026Covering Human Action Space for Computer Use: Data Synthesis and Benchmark
  8. 2026GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents