mlsanityData to 14 Oct 2026

MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents

arXiv 2507.19478 ↗First posted 25 Jul 2025cs.CV cs.CLcited by 67

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

We introduce MMBench-GUI, a hierarchical benchmark for evaluating GUI automation agents across Windows, macOS, Linux, iOS, Android, and Web platforms. It comprises four levels: GUI Content Understanding, Element Grounding, Task Automation, and Task Collaboration, covering essential skills for GUI agents. In addition, we propose a novel Efficiency-Quality Area (EQA) metric to assess GUI agent execution efficiency in online automation scenarios. Through MMBench-GUI, we identify accurate visual grounding as a critical determinant of overall task success, emphasizing the substantial benefits of modular frameworks that integrate specialized grounding modules. Furthermore, to achieve reliable GUI automation, an agent requires strong task planning and cross-platform generalization abilities, with long-context memory, a broad action space, and long-term reasoning playing a critical role. More important, task efficiency remains a critically underexplored dimension, and all models suffer from substantial inefficiencies, with excessive redundant steps even when tasks are ultimately completed. The integration of precise localization, effective planning, and early stopping strategies is indispensable to enable truly efficient and scalable GUI automation. Our benchmark code, evaluation data, and running environment will be publicly available at https://github.com/open-compass/MMBench-GUI.

Key facts. Cited by 67 papers in the map (data through 14 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 48 of its 54 references that are in the map. First posted on arXiv 25 Jul 2025.

Builds on

48 of 54 references.

Closest ancestors · 19

Same direction, newest first

  1. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents3 Jun
  2. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis19 May
  3. 2025Breaking the Data Barrier -- Building GUI Agents Through Task Generalization14 Apr
  4. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use4 Apr
  5. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
  6. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis27 Dec
  7. 2024AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents31 Oct
  8. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
  9. 2024AutoGLM: Autonomous Foundation Agents for GUIs28 Oct
  10. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
  11. 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale12 Sep
  12. 2024GUICourse: From General Vision Language Models to Versatile GUI Agents17 Jun
7 more
  1. 2024On the Effects of Data Scale on UI Control Agents6 Jun
  2. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments11 Apr
  3. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web27 Feb
  4. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvement12 Feb
  5. 2024ScreenAgent: A Vision Language Model-driven Computer Control Agent9 Feb
  6. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan
  7. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec

Other references · 29

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2025Qwen2.5-VL Technical Reportcited by 5,587
  3. 2024GPT-4o System Cardcited by 4,588
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  7. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  8. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  9. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  10. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  11. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  12. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
17 more
  1. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
  2. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 438
  3. 2025Kimi-VL Technical Reportcited by 388
  4. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 339
  5. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 319
  6. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 278
  7. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 232
  8. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 194
  9. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 160
  10. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 140
  11. 2024VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agentscited by 103
  12. 2021WebSRC: A Dataset for Web-Based Structural Reading Comprehensioncited by 100
  13. 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 77
  14. 2022ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshotscited by 66
  15. 2025ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflowscited by 52
  16. 2023ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automationcited by 45
  17. 2025UI-E2I-Synth: Advancing GUI Grounding with Large-Scale Instruction Synthesiscited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 23; 2026: 44.

67 papers.

Sort:

By area:

  1. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  2. 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agentscited by 91
  3. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 60
  4. 2025Are Video Models Ready as Zero-Shot Reasoners? An Empirical Study with the MME-CoF Benchmarkcited by 43
  5. 2026UI-Venus-1.5 Technical Reportcited by 41
  6. 2025Step-GUI Technical Reportcited by 39
  7. 2025InfiGUI-G1: Advancing GUI Grounding with Adaptive Exploration Policy Optimizationcited by 27
  8. 2026InfiniteWeb: Scalable Web Environment Synthesis for GUI Agent Trainingcited by 27
  9. 2025UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoningcited by 22
  10. 2025Grounding Computer Use Agents on Human Demonstrationscited by 22
  11. 2026MMSkills: Towards Multimodal Skills for General Visual Agentscited by 16
  12. 2025VeriOS: Query-Driven Proactive Human-Agent-GUI Interaction for Trustworthy OS Agentscited by 15
38 more
  1. 2026ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agentscited by 14
  2. 2025GhostEI-Bench: Do Mobile Agents Resilience to Environmental Injection in Dynamic On-Device Environments?cited by 12
  3. 2026Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Groundingcited by 11
  4. 2026Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agentscited by 11
  5. 2026AndroTMem: From Interaction Trajectories to Anchored Memory in Long-Horizon GUI Agentscited by 10
  6. 2026OS-Themis: A Scalable Critic Framework for Generalist GUI Rewardscited by 10
  7. 2026AndroidDaily: A Verifiable Benchmark for Mobile GUI Agents on Real-World Closed-Source Applicationscited by 10
  8. 2025GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Groundingcited by 8
  9. 2025SCUBA: Salesforce Computer Use Benchmarkcited by 7
  10. 2025VenusBench-GD: A Comprehensive Multi-Platform GUI Benchmark for Diverse Grounding Taskscited by 7
  11. 2025OS-Oracle: A Comprehensive Framework for Cross-Platform GUI Critic Modelscited by 6
  12. 2026ScreenParse: Moving Beyond Sparse Grounding with Complete Screen Parsing Supervisioncited by 6
  13. 2025From Exploration to Exploitation: A Two-Stage Entropy RLVR Approach for Noise-Tolerant MLLM Trainingcited by 5
  14. 2025Using GUI Agent for Electronic Design Automationcited by 5
  15. 2026MME-CoF-Pro: Evaluating Reasoning Coherence in Video Generative Models with Text and Visual Hintscited by 5
  16. 2026Measure Twice, Click Once: Co-evolving Proposer and Visual Critic via Reinforcement Learning for GUI Groundingcited by 5
  17. 2025Generalist Scanner Meets Specialist Locator: A Synergistic Coarse-to-Fine Framework for Robust GUI Groundingcited by 4
  18. 2025Say One Thing, Do Another? Diagnosing Reasoning-Execution Gaps in VLM-Powered Mobile-Use Agentscited by 4
  19. 2026POINTS-GUI-G: GUI-Grounding Journeycited by 4
  20. 2026WebTestPilot: Agentic End-to-End Web Testing against Natural Language Specification by Inferring Oracles with Symbolized GUI Elementscited by 4
  21. 2026What's Missing in Screen-to-Action? Towards a UI-in-the-Loop Paradigm for Multimodal GUI Reasoningcited by 4
  22. 2026Weblica: Scalable and Reproducible Training Environments for Visual Web Agentscited by 4
  23. 2026What Happens Before Decoding? Prefill Determines GUI Grounding in VLMscited by 4
  24. 2026Trust the Right Teacher: Quality-Aware Self-Distillation for GUI Groundingcited by 4
  25. 2026HyMobileAgent: Data-Environment Co-Scaling for Efficient GUI Agentscited by 4
  26. 2025GUI Knowledge Bench: Revealing the Knowledge Gap of VLMs in GUI Taskscited by 3
  27. 2025Enhancing Trustworthy GUI Grounding via Self-Critiqued Reinforcement Learningcited by 3
  28. 2025UI-CUBE: Enterprise-Grade Computer Use Agent Benchmarking Beyond Task Accuracy to Operational Reliabilitycited by 3
  29. 2026GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agentscited by 3
  30. 2026MedSPOT: A Workflow-Aware Sequential Grounding Benchmark for Clinical GUIcited by 3
  31. 2026VISTA: View-Consistent Self-Verified Training for GUI Groundingcited by 3
  32. 2026EvoCUA-1.5: Online Reinforcement Learning for Multi-turn Computer-Use Agentscited by 3
  33. 2026Interactive Reward Agent: GUI Task Evaluation via Environment-State Verificationcited by 3
  34. 2025GUIrilla: A Scalable Framework for Automated Desktop UI Explorationcited by 2
  35. 2025DaMo: Data Mixing Optimizer in Fine-tuning Multimodal LLMs for Mobile Phone Agentscited by 2
  36. 2026One Forward Beats Two: InnerZoom for Accurate and Efficient GUI Groundingcited by 2
  37. 2026Test-Time Self-Evolving GUI Visual Grounding via Reflection-Guided On-Policy Self-Distillationcited by 2
  38. 2026AutoGUI-v2: A Comprehensive Multi-Modal GUI Functionality Understanding Benchmarkcited by 1

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025ProBench: Benchmarking GUI Agents with Accurate Process Information
  2. 2025WorldGUI: An Interactive Benchmark for Desktop GUI Automation from Any Starting Point
  3. 2026WindowsWorld: A Process-Centric Benchmark of Autonomous GUI Agents in Professional Cross-Application Environments
  4. 2026MobileBench-OL: A Comprehensive Chinese Benchmark for Evaluating Mobile GUI Agents in Real-World Environment
  5. 2026Beyond Sequential Interaction: Benchmarking Parallel Execution and Coordination for GUI Agents
  6. 2026VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnostics
  7. 2026JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition
  8. 2025MAS-Bench: A Unified Benchmark for Shortcut-Augmented Hybrid Mobile GUI Agents