mlsanityData to 14 Oct 2026

ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use

arXiv 2504.07981 ↗First posted 4 Apr 2025cs.CV cs.HC cs.MMcited by 254

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Recent advancements in Multi-modal Large Language Models (MLLMs) have led to significant progress in developing GUI agents for general tasks such as web browsing and mobile phone use. However, their application in professional domains remains under-explored. These specialized workflows introduce unique challenges for GUI perception models, including high-resolution displays, smaller target sizes, and complex environments. In this paper, we introduce ScreenSpot-Pro, a new benchmark designed to rigorously evaluate the grounding capabilities of MLLMs in high-resolution professional settings. The benchmark comprises authentic high-resolution images from a variety of professional domains with expert annotations. It spans 23 applications across five industries and three operating systems. Existing GUI grounding models perform poorly on this dataset, with the best model achieving only 18.9%. Our experiments reveal that strategically reducing the search area enhances accuracy. Based on this insight, we propose ScreenSeekeR, a visual search method that utilizes the GUI knowledge of a strong planner to guide a cascaded search, achieving state-of-the-art performance with 48.1% without any additional training. We hope that our benchmark and findings will advance the development of GUI agents for professional applications. Code, data and leaderboard can be found at https://gui-agent.github.io/grounding-leaderboard.

Key facts. Cited by 254 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 29 references that are in the map. First posted on arXiv 4 Apr 2025.

Builds on

24 of 29 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Improved GUI Grounding via Iterative Narrowing18 Nov
  2. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
  3. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
  4. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs8 Apr
  5. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec

Other references · 19

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  3. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  6. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  7. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  8. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  9. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  10. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 567
  11. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
  12. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353
7 more
  1. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 232
  2. 2024Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Modelscited by 161
  3. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 148
  4. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 140
  5. 2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?cited by 100
  6. 2022A data-driven approach for learning to control computerscited by 88
  7. 2024VGA: Vision GUI Assistant -- Minimizing Hallucinations through Image-Centric Fine-Tuningcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 97; 2026: 157.

254 papers.

Sort:

By area:

  1. 2025Qwen3-VL Technical Reportcited by 2,793
  2. 2025Kimi-VL Technical Reportcited by 388
  3. 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agentscited by 244
  4. 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 160
  5. 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasonerscited by 140
  6. 2025MiMo-VL Technical Reportcited by 115
  7. 2025GTA1: GUI Test-time Scaling Agentcited by 106
  8. 2025MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Serverscited by 96
  9. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 95
  10. 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agentscited by 91
  11. 2025Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learningcited by 81
  12. 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agentscited by 74
38 more
  1. 2025Detect Anything via Next Point Predictioncited by 73
  2. 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 69
  3. 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 67
  4. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 60
  5. 2025FineVision: Open Data Is All You Needcited by 53
  6. 2026EvoCUA: Evolving Computer Use Agents via Learning from Scalable Synthetic Experiencecited by 53
  7. 2026Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perceptioncited by 50
  8. 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 50
  9. 2025AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuningcited by 44
  10. 2025Are Video Models Ready as Zero-Shot Reasoners? An Empirical Study with the MME-CoF Benchmarkcited by 43
  11. 2025TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agentscited by 42
  12. 2026OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Taskscited by 40
  13. 2025Step-GUI Technical Reportcited by 39
  14. 2025CoAct-1: Computer-using Multi-Agent System with Coding Actionscited by 35
  15. 2025Fara-7B: An Efficient Agentic Model for Computer Usecited by 35
  16. 2025UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Groundingcited by 33
  17. 2025UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learningcited by 32
  18. 2026STEP3-VL-10B Technical Reportcited by 32
  19. 2025Visual Test-time Scaling for GUI Agent Groundingcited by 30
  20. 2025One RL to See Them All: Visual Triple Unified Reinforcement Learningcited by 30
  21. 2026Workspace-Bench 1.0: Benchmarking AI Agents on Workspace Tasks with Large-Scale File Dependenciescited by 29
  22. 2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 28
  23. 2025InfiGUI-G1: Advancing GUI Grounding with Adaptive Exploration Policy Optimizationcited by 27
  24. 2026InfiniteWeb: Scalable Web Environment Synthesis for GUI Agent Trainingcited by 27
  25. 2025Phi-Ground Tech Report: Advancing Perception in GUI Groundingcited by 25
  26. 2025Game-TARS: Pretrained Foundation Models for Scalable Generalist Multimodal Game Agentscited by 24
  27. 2025A Survey on Agentic Multimodal Large Language Modelscited by 23
  28. 2025UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoningcited by 22
  29. 2026AutoWebWorld: Synthesizing Infinite Verifiable Web Environments via Finite State Machinescited by 21
  30. 2026Odysseys: Benchmarking Web Agents on Realistic Long Horizon Taskscited by 21
  31. 2025Auto-scaling Continuous Memory for GUI Agentcited by 20
  32. 2025MagicGUI: A Foundational Mobile GUI Agent with Scalable Data Pipeline and Reinforcement Fine-tuningcited by 19
  33. 2025Can World Simulators Reason? Gen-ViRe: A Generative Visual Reasoning Benchmarkcited by 19
  34. 2026MolmoWeb: Open Visual Web Agent and Open Data for the Open Webcited by 19
  35. 2025NVIDIA Nemotron Nano V2 VLcited by 17
  36. 2025Ferret-UI Lite: Lessons from Building Small On-Device GUI Agentscited by 16
  37. 2026MMSkills: Towards Multimodal Skills for General Visual Agentscited by 16
  38. 2025VeriOS: Query-Driven Proactive Human-Agent-GUI Interaction for Trustworthy OS Agentscited by 15

50 most cited of 254 All 254 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Generalist Scanner Meets Specialist Locator: A Synergistic Coarse-to-Fine Framework for Robust GUI Grounding
  2. 2026GoClick: Lightweight Element Grounding Model for Autonomous GUI Interaction
  3. 2025DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoning
  4. 2025Chain-of-Ground: Improving GUI Grounding via Iterative Reasoning and Reference Feedback
  5. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis
  6. 2025R-VLM: Region-Aware Vision Language Model for Precise GUI Grounding
  7. 2024Ponder & Press: Advancing Visual GUI Agent towards General Computer Control
  8. 2024MobileViews: A Million-scale and Diverse Mobile GUI Dataset