# ProBench: Benchmarking GUI Agents with Accurate Process Information

arXiv: https://arxiv.org/abs/2511.09157 · HTML: https://mlsanity.com/paper/2511.09157/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use.
- Builds on 20 of its 34 references that are in the map.
- First posted on arXiv 12 Nov 2025.

## Placed in

- LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/)

## Builds on (most cited first, 20 of 20)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025)
- [AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents](https://mlsanity.com/paper/2405.14573/) (2024)
- [OS-ATLAS: A Foundation Action Model for Generalist GUI Agents](https://mlsanity.com/paper/2410.23218/) (2024)
- [Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V](https://mlsanity.com/paper/2310.11441/) (2023)
- [GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents](https://mlsanity.com/paper/2504.10458/) (2025)
- [UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learning](https://mlsanity.com/paper/2503.21620/) (2025)
- [Android in the Zoo: Chain-of-Action-Thought for GUI Agents](https://mlsanity.com/paper/2403.02713/) (2024)
- [AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents](https://mlsanity.com/paper/2410.24024/) (2024)
- [ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshots](https://mlsanity.com/paper/2209.08199/) (2022)
- [Benchmarking Mobile Device Control Agents across Diverse Configurations](https://mlsanity.com/paper/2404.16660/) (2024)
- [UGIF: UI Grounded Instruction Following](https://mlsanity.com/paper/2211.07615/) (2022)
- [A3: Android Agent Arena for Mobile GUI Agents with Essential-State Procedural Evaluation](https://mlsanity.com/paper/2501.01149/) (2025)

## Built on by (most cited first, 15 of 15)

- [MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments](https://mlsanity.com/paper/2512.19432/) (2025)
- [Hybrid Self-evolving Structured Memory for GUI Agents](https://mlsanity.com/paper/2603.10291/) (2026)
- [MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research](https://mlsanity.com/paper/2605.26114/) (2026)
- [WindowsWorld: A Process-Centric Benchmark of Autonomous GUI Agents in Professional Cross-Application Environments](https://mlsanity.com/paper/2604.27776/) (2026)
- [What Memory Do GUI Agents Really Need? From Passive Records to Active Task-Driving States](https://mlsanity.com/paper/2606.31612/) (2026)
- [Towards Scalable Lightweight GUI Agents via Multi-role Orchestration](https://mlsanity.com/paper/2604.13488/) (2026)
- [PAGER: Bridging the Semantic-Execution Gap in Point-Precise Geometric GUI Control](https://mlsanity.com/paper/2605.15963/) (2026)
- [ChartAct: A Benchmark for Dynamic Chart Understanding](https://mlsanity.com/paper/2605.26994/) (2026)
- [OS-SPEAR: A Toolkit for the Safety, Performance,Efficiency, and Robustness Analysis of OS Agents](https://mlsanity.com/paper/2604.24348/) (2026)
- [Beyond Binary: Reframing GUI Critique as Continuous Semantic Alignment](https://mlsanity.com/paper/2605.14311/) (2026)
- [ElderBench: Benchmarking Autonomous Mobile Agents for Older Adults](https://mlsanity.com/paper/2609.04850/) (2026)
- [MATE: Policy-Aware Security Auditing for Mobile Agents via Synthesis-Driven Trajectory Learning](https://mlsanity.com/paper/2609.22724/) (2026)
- [VibeMemBench: Evaluating Memory Systems for Coding Agents on Real Repository Coding Tasks](https://mlsanity.com/paper/2609.23570/) (2026)
- [GUITAR: Structured Failure Diagnosis of GUI Agents via State Transitions](https://mlsanity.com/paper/2609.34113/) (2026)
- [How GUI Agents Fail: GraceBench for Execution Stability Diagnosis](https://mlsanity.com/paper/2610.19900/) (2026)

## Abstract (arXiv)

With the deep integration of artificial intelligence and interactive technology, Graphical User Interface (GUI) Agent, as the carrier connecting goal-oriented natural language and real-world devices, has received widespread attention from the community. Contemporary benchmarks aim to evaluate the comprehensive capabilities of GUI agents in GUI operation tasks, generally determining task completion solely by inspecting the final screen state. However, GUI operation tasks consist of multiple chained steps while not all critical information is presented in the final few pages. Although a few research has begun to incorporate intermediate steps into evaluation, accurately and automatically capturing this process information still remains an open challenge. To address this weakness, we introduce ProBench, a comprehensive mobile benchmark with over 200 challenging GUI tasks covering widely-used scenarios. Remaining the traditional State-related Task evaluation, we extend our dataset to include Process-related Task and design a specialized evaluation method. A newly introduced Process Provider automatically supplies accurate process information, enabling presice assessment of agent's performance. Our evaluation of advanced GUI agents reveals significant limitations for real-world GUI scenarios. These shortcomings are prevalent across diverse models, including both large-scale generalist models and smaller, GUI-specific models. A detailed error analysis further exposes several universal problems, outlining concrete directions for future improvements.