mlsanityData to 14 Oct 2026

Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis

arXiv 2505.13227 ↗First posted 19 May 2025cs.AI cs.CL cs.CVcited by 139

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

Graphical user interface (GUI) grounding, the ability to map natural language instructions to specific actions on graphical user interfaces, remains a critical bottleneck in computer use agent development. Current benchmarks oversimplify grounding tasks as short referring expressions, failing to capture the complexity of real-world interactions that require software commonsense, layout understanding, and fine-grained manipulation capabilities. To address these limitations, we introduce OSWorld-G, a comprehensive benchmark comprising 564 finely annotated samples across diverse task types including text matching, element recognition, layout understanding, and precise manipulation. Additionally, we synthesize and release the largest computer use grounding dataset Jedi, which contains 4 million examples through multi-perspective decoupling of tasks. Our multi-scale models trained on Jedi demonstrate its effectiveness by outperforming existing approaches on ScreenSpot-v2, ScreenSpot-Pro, and our OSWorld-G. Furthermore, we demonstrate that improved grounding with Jedi directly enhances agentic capabilities of general foundation models on complex computer tasks, improving from 5% to 27% on OSWorld. Through detailed ablation studies, we identify key factors contributing to grounding performance and verify that combining specialized data for different interface elements enables compositional generalization to novel interfaces. All benchmark, data, checkpoints, and code are open-sourced and available at https://osworld-grounding.github.io.

Key facts. Cited by 139 papers in the map (data through 14 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 40 of its 53 references that are in the map. First posted on arXiv 19 May 2025.

Builds on

40 of 53 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025PC-Agent: A Hierarchical Multi-Agent Collaboration Framework for Complex Task Automation on PC20 Feb
  2. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis27 Dec
  3. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
  4. 2024Agent S: An Open Agentic Framework that Uses Computers Like a Human10 Oct
  5. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
  6. 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale12 Sep
  7. 2024GUICourse: From General Vision Language Models to Versatile GUI Agents17 Jun
  8. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning14 Jun
  9. 2024On the Effects of Data Scale on UI Control Agents6 Jun
  10. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agents5 Mar
  11. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web27 Feb
  12. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan
1 more
  1. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec

Other references · 27

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  2. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  3. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  4. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  5. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  6. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  7. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  8. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  9. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  10. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  11. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  12. 2025Seed1.5-VL Technical Reportcited by 373
15 more
  1. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 363
  2. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 319
  3. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 284
  4. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 278
  5. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 268
  6. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 256
  7. 2024OmniParser for Pure Vision Based GUI Agentcited by 198
  8. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 181
  9. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 140
  10. 2020Widget Captioning: Generating Natural Language Description for Mobile User Interface Elementscited by 95
  11. 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 95
  12. 2021AndroidEnv: A Reinforcement Learning Platform for Androidcited by 90
  13. 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 77
  14. 2023WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semanticscited by 47
  15. 2025OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Modelscited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 52; 2026: 87.

139 papers.

Sort:

By area:

  1. 2025Qwen3-VL Technical Reportcited by 2,793
  2. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  3. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 190
  4. 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 160
  5. 2025MiMo-VL Technical Reportcited by 115
  6. 2026Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forwardcited by 113
  7. 2025GTA1: GUI Test-time Scaling Agentcited by 106
  8. 2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 93
  9. 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agentscited by 91
  10. 2026Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agentscited by 79
  11. 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 67
  12. 2025GUI-G2: Gaussian Reward Modeling for GUI Groundingcited by 63
38 more
  1. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 60
  2. 2026EvoCUA: Evolving Computer Use Agents via Learning from Scalable Synthetic Experiencecited by 53
  3. 2025ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agentscited by 50
  4. 2026Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perceptioncited by 50
  5. 2026UI-Venus-1.5 Technical Reportcited by 41
  6. 2026LongCat-Next: Lexicalizing Modalities as Discrete Tokenscited by 36
  7. 2025CoAct-1: Computer-using Multi-Agent System with Coding Actionscited by 35
  8. 2025UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Groundingcited by 33
  9. 2026STEP3-VL-10B Technical Reportcited by 32
  10. 2025UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Actioncited by 31
  11. 2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 28
  12. 2026CUA-Gym: Scaling Verifiable Training Environments and Tasks for Computer-Use Agentscited by 23
  13. 2025VideoAgentTrek: Computer Use Pretraining from Unlabeled Videoscited by 22
  14. 2025UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoningcited by 22
  15. 2025Grounding Computer Use Agents on Human Demonstrationscited by 22
  16. 2026OS-Symphony: A Holistic Framework for Robust and Generalist Computer-Using Agentcited by 21
  17. 2026Odysseys: Benchmarking Web Agents on Realistic Long Horizon Taskscited by 21
  18. 2025How Do AI Agents Do Human Work? Comparing AI and Human Workflows Across Diverse Occupationscited by 19
  19. 2025CUARewardBench: A Benchmark for Evaluating Reward Models on Computer-using Agentcited by 18
  20. 2025MGA: Memory-Driven GUI Agent for Observation-Centric Interactioncited by 18
  21. 2025Ferret-UI Lite: Lessons from Building Small On-Device GUI Agentscited by 16
  22. 2026GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RLcited by 16
  23. 2026MolmoPoint: Better Pointing for VLMs with Grounding Tokenscited by 15
  24. 2026ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agentscited by 14
  25. 2025GUI-ReWalk: Massive Data Generation for GUI Agent via Stochastic Exploration and Intent-Aware Reasoningcited by 13
  26. 2025Environment Scaling for Interactive Agentic Experience Collection: A Surveycited by 13
  27. 2026OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Executioncited by 13
  28. 2025Learning GUI Grounding with Spatial Reasoning from Visual Feedbackcited by 12
  29. 2025R-WoM: Retrieval-augmented World Model For Computer-use Agentscited by 12
  30. 2025MVP: Multiple View Prediction Improves GUI Groundingcited by 12
  31. 2026The Blind Spot of Agent Safety: How Benign User Instructions Expose Critical Vulnerabilities in Computer-Use Agentscited by 12
  32. 2026UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experiencecited by 11
  33. 2026Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Groundingcited by 11
  34. 2026Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agentscited by 11
  35. 2025Towards Understanding Visual Grounding in Visual Language Modelscited by 10
  36. 2025GUI-ARP: Enhancing Grounding with Adaptive Region Perception for GUI Agentscited by 10
  37. 2025textscGUI-Spotlight: Adaptive Iterative Focus Refinement for Enhanced GUI Visual Groundingcited by 10
  38. 2026FocusUI: Efficient UI Grounding via Position-Preserving Visual Token Selectioncited by 10

50 most cited of 139 All 139 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Generalist Scanner Meets Specialist Locator: A Synergistic Coarse-to-Fine Framework for Robust GUI Grounding
  2. 2026AdaZoom-GUI: Adaptive Zoom-based GUI Grounding with Instruction Refinement
  3. 2026WinDeskGround: A Benchmark for Robust GUI Grounding in Complex Multi-Window Desktop Environments
  4. 2025ReGUIDE: Data Efficient GUI Grounding via Spatial Reasoning and Search
  5. 2026Hallucination-Free GUI Grounding via Regression-Free Layout-Aware Matching
  6. 2026GoClick: Lightweight Element Grounding Model for Autonomous GUI Interaction
  7. 2025GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration
  8. 2025GUIrilla: A Scalable Framework for Automated Desktop UI Exploration