PAGER: Bridging the Semantic-Execution Gap in Point-Precise Geometric GUI Control
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
also in 1 more
Abstract (arXiv)
Large vision-language models have significantly advanced GUI agents, enabling executable interaction across web, mobile, and desktop interfaces. Yet these gains largely rely on a forgiving region-tolerant paradigm, where many nearby pixels inside the same component remain valid. Precise geometric construction breaks this assumption: actions must land on points in continuous canvas space rather than tolerant regions. Because geometric primitives carry ontological dependencies, a local coordinate error can induce cascading topological failures that distort downstream objects and invalidate the final construction. We identify this regime as precision-sensitive GUI tasks, requiring point-level accuracy, geometry-aware verification, and robustness to dependency-driven error propagation. To benchmark it, we introduce PAGE Bench, with 4,906 problems and over 224K process-supervised, pixel-level GUI actions. We further propose PAGER, a topology-aware agent that decomposes construction into dependency-structured planning and pixel-level execution. Pixel-grounded supervised tuning establishes executable action grammar, while precision-aligned reinforcement learning mitigates rollout-induced exposure bias through state-conditioned geometric feedback. Experiments reveal a pronounced Semantic-Execution Gap: general multimodal models can exceed 88% action type accuracy yet remain below 6% task success. PAGER closes this gap, delivering 4.1x higher task success than the strongest evaluated general baseline and raising step success rate from below 9% for GUI-specialized agents to over 62%, establishing a new state of the art for point-precise GUI control.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 58 references that are in the map. First posted on arXiv 15 May 2026.
Builds on
44 of 58 references.
Closest ancestors · 18
Same direction, newest first
- 2026MedSPOT: A Workflow-Aware Sequential Grounding Benchmark for Clinical GUI20 Mar
- 2026Milestones over Outcome: Unlocking Geometric Reasoning with Sub-Goal Verifiable Reward8 Jan
- 2025History-Aware Reasoning for GUI Agents12 Nov
- 2025GeoFM: Enhancing Geometric Reasoning of MLLMs via Synthetic Data Generation through Formal Language31 Oct
- 2025GeoVLMath: Enhancing Geometry Reasoning in Vision-Language Models via Cross-Modal Reward for Auxiliary Line Creation13 Oct
- 2025GeoSketch: A Neural-Symbolic Approach to Geometric Multimodal Reasoning with Auxiliary Line Construction and Affine Transformation26 Sep
- 2025OpenCUA: Open Foundations for Computer-Use Agents12 Aug
- 2025UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Grounding29 Jul
- 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning1 Jul
- 2025ReGUIDE: Data Efficient GUI Grounding via Spatial Reasoning and Search21 May
- 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasoners19 Apr
- 2025GeoSense: Evaluating Identification and Application of Geometric Principles in Multimodal Reasoning17 Apr
6 more
- 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents14 Apr
- 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization17 Mar
- 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
- 2024GUICourse: From General Vision Language Models to Versatile GUI Agents17 Jun
- 2024CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation19 Feb
- 2023CogAgent: A Visual Language Model for GUI Agents14 Dec
Other references · 26
- 2025Qwen3-VL Technical Reportcited by 2,793
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
- 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
- 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 647
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 610
- 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 572
- 2025Kimi-VL Technical Reportcited by 388
- 2024GUI Agents: A Surveycited by 103
- 2025Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learningcited by 81
- 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agentscited by 74
- 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflectioncited by 67
- 2025MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environmentcited by 52
14 more
- 2025MobileRL: Online Agentic Reinforcement Learning for Mobile GUI Agentscited by 43
- 2025MCPToolBench++: A Large Scale AI Agent Model Context Protocol MCP Tool Use Benchmarkcited by 32
- 2025MM-PRM: Enhancing Multimodal Mathematical Reasoning with Scalable Step-Level Supervisioncited by 31
- 2025WorldGUI: An Interactive Benchmark for Desktop GUI Automation from Any Starting Pointcited by 20
- 2025SOLIDGEO: Measuring Multimodal Spatial Math Reasoning in Solid Geometrycited by 18
- 2026GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RLcited by 16
- 2025ProBench: Benchmarking GUI Agents with Accurate Process Informationcited by 15
- 2025MATP-BENCH: Can MLLM Be a Good Automated Theorem Prover for Multimodal Problems?cited by 13
- 2025Geoint-R1: Formalizing Multimodal Geometric Reasoning with Dynamic Auxiliary Constructionscited by 11
- 2025GUI-360^circ: A Comprehensive Dataset and Benchmark for Computer-Using Agentscited by 10
- 2024VisAidMath: Benchmarking Visual-Aided Mathematical Reasoningcited by 9
- 2025GeoBench: Rethinking Multimodal Geometric Problem-Solving via Hierarchical Evaluationcited by 4
- 2025Beyond Clicking:A Step Towards Generalist GUI Grounding via Text Draggingcited by 3
- 2026GeoChallenge: A Multi-Answer Multiple-Choice Benchmark for Geometric Reasoning with Diagramscited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026GUI-Primitives: Diagnosing Spatial Reasoning Failures in Vision-Language GUI Grounding
- 2026HATS: Hardness-Aware Trajectory Synthesis for GUI Agents
- 2026Learn from Move: the Next Step for GUI Agents
- 2024Ponder & Press: Advancing Visual GUI Agent towards General Computer Control
- 2025Breaking the Data Barrier -- Building GUI Agents Through Task Generalization
- 2026Measure Twice, Click Once: Co-evolving Proposer and Visual Critic via Reinforcement Learning for GUI Grounding
- 2026ActionEngine: From Reactive to Programmatic GUI Agents via State Machine Memory
- 2026GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents