Data to 14 Oct 2026

UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction

arXiv 2503.15661 ↗First posted 19 Mar 2025cs.CV cs.AI cs.CLcited by 77

Not on the map. Why?

Cited by 48 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Autonomous agents that navigate Graphical User Interfaces (GUIs) to automate tasks like document editing and file management can greatly enhance computer workflows. While existing research focuses on online settings, desktop environments, critical for many professional and everyday tasks, remain underexplored due to data collection challenges and licensing issues. We introduce UI-Vision, the first comprehensive, license-permissive benchmark for offline, fine-grained evaluation of computer use agents in real-world desktop environments. Unlike online benchmarks, UI-Vision provides: (i) dense, high-quality annotations of human demonstrations, including bounding boxes, UI labels, and action trajectories (clicks, drags, and keyboard inputs) across 83 software applications, and (ii) three fine-to-coarse grained tasks-Element Grounding, Layout Grounding, and Action Prediction-with well-defined metrics to rigorously evaluate agents' performance in desktop environments. Our evaluation reveals critical limitations in state-of-the-art models like UI-TARS-72B, including issues with understanding professional software, spatial reasoning, and complex actions like drag-and-drop. These findings highlight the challenges in developing fully autonomous computer use agents. By releasing UI-Vision as open-source, we aim to advance the development of more capable agents for real-world desktop tasks.

Key facts. Cited by 77 papers in the map (data through 14 Oct 2026). Builds on 41 of its 56 references that are in the map. First posted on arXiv 19 Mar 2025.

Builds on

41 of 56 references.

References · 41

2022Training language models to follow instructions with human feedbackcited by 15,971
2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079

29 more

2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
2023CogAgent: A Visual Language Model for GUI Agentscited by 704
2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 610
2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 567
2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 382
2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 378
2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 366
2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 363
2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 319
2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 299
2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 248
2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 232
2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scalecited by 223
2024OmniParser for Pure Vision Based GUI Agentcited by 198
2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 181
2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 180
2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 176
2024Android in the Zoo: Chain-of-Action-Thought for GUI Agentscited by 159
2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 157
2024Aria-UI: Visual Grounding for GUI Instructionscited by 140
2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learncited by 97
2020Widget Captioning: Generating Natural Language Description for Mobile User Interface Elementscited by 95
2021AndroidEnv: A Reinforcement Learning Platform for Androidcited by 90
2024Ferret-UI 2: Mastering Universal User Interface Understanding Across Platformscited by 51
2023WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semanticscited by 47
2024VideoGUI: A Benchmark for GUI Automation from Instructional Videoscited by 35
2024Visual Grounding Methods for Efficient Interaction with Desktop Graphical User Interfacescited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 27; 2026: 50.

77 papers.

Sort:

By area:

2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 190
2025OpenCUA: Open Foundations for Computer-Use Agentscited by 160
2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 139
2025MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Serverscited by 96
2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 93
2025Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learningcited by 81
2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 67
2025Paper2Poster: Towards Multimodal Poster Automation from Scientific Paperscited by 63
2025A Survey on (M)LLM-Based GUI Agentscited by 62
2025TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agentscited by 42
2026UI-Venus-1.5 Technical Reportcited by 41
2026Code2World: A GUI World Model via Renderable Code Generationcited by 29

38 more

2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 28
2025InfiGUI-G1: Advancing GUI Grounding with Adaptive Exploration Policy Optimizationcited by 27
2025Phi-Ground Tech Report: Advancing Perception in GUI Groundingcited by 25
2025A Survey on Agentic Multimodal Large Language Modelscited by 23
2025Grounding Computer Use Agents on Human Demonstrationscited by 22
2026ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agentscited by 14
2025GUI-ReWalk: Massive Data Generation for GUI Agent via Stochastic Exploration and Intent-Aware Reasoningcited by 13
2025MVP: Multiple View Prediction Improves GUI Groundingcited by 12
2026Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Groundingcited by 11
2026Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agentscited by 11
2025GUI-ARP: Enhancing Grounding with Adaptive Region Perception for GUI Agentscited by 10
2025textscGUI-Spotlight: Adaptive Iterative Focus Refinement for Enhanced GUI Visual Groundingcited by 10
2025GUI-360^circ: A Comprehensive Dataset and Benchmark for Computer-Using Agentscited by 10
2026FocusUI: Efficient UI Grounding via Position-Preserving Visual Token Selectioncited by 10
2025Reinforcement Learning for Large Model: A Surveycited by 9
2025Computer-Use Agents as Judges for Generative User Interfacecited by 9
2025GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Groundingcited by 8
2026UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Groundingcited by 8
2025VenusBench-GD: A Comprehensive Multi-Platform GUI Benchmark for Diverse Grounding Taskscited by 7
2025OS-MAP: How Far Can Computer-Using Agents Go in Breadth and Depth?cited by 6
2026Zoom to Essence: Trainless GUI Grounding by Inferring upon Interface Elementscited by 5
2026CUA-Suite: Massive Human-annotated Video Demonstrations for Computer-Use Agentscited by 5
2026Measure Twice, Click Once: Co-evolving Proposer and Visual Critic via Reinforcement Learning for GUI Groundingcited by 5
2026FineState-Bench: Benchmarking State-Conditioned Grounding for Fine-grained GUI State Settingcited by 5
2025Improving GUI Grounding with Explicit Position-to-Coordinate Mappingcited by 4
2026POINTS-GUI-G: GUI-Grounding Journeycited by 4
2026WebTestBench: Evaluating Computer-Use Agents towards End-to-End Automated Web Testingcited by 4
2026GUIDE: A Benchmark for Understanding and Assisting Users in Open-Ended GUI Taskscited by 4
2026What's Missing in Screen-to-Action? Towards a UI-in-the-Loop Paradigm for Multimodal GUI Reasoningcited by 4
2026What Happens Before Decoding? Prefill Determines GUI Grounding in VLMscited by 4
2026Trust the Right Teacher: Quality-Aware Self-Distillation for GUI Groundingcited by 4
2025The Stackelberg Speaker: Optimizing Persuasive Communication in Social Deduction Gamescited by 3
2025Beyond Clicking:A Step Towards Generalist GUI Grounding via Text Draggingcited by 3
2026GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agentscited by 3
2026CAPTCHA Solving for Native GUI Agents: Automated Reasoning-Action Data Generation and Self-Corrective Trainingcited by 3
2026GUI-Perturbed: Domain Randomization Reveals Systematic Brittleness in GUI Grounding Modelscited by 3
2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitantscited by 3
2026Plover: Steering GUI Agents through Plan-Centric Interactioncited by 3

50 most cited of 77 All 77 (JSON)

Similar, not cited

Close in wording, no citation either way