mlsanityData to 14 Oct 2026

PAGER: Bridging the Semantic-Execution Gap in Point-Precise Geometric GUI Control

arXiv 2605.15963 ↗First posted 15 May 2026cs.AIcited by 1

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Large vision-language models have significantly advanced GUI agents, enabling executable interaction across web, mobile, and desktop interfaces. Yet these gains largely rely on a forgiving region-tolerant paradigm, where many nearby pixels inside the same component remain valid. Precise geometric construction breaks this assumption: actions must land on points in continuous canvas space rather than tolerant regions. Because geometric primitives carry ontological dependencies, a local coordinate error can induce cascading topological failures that distort downstream objects and invalidate the final construction. We identify this regime as precision-sensitive GUI tasks, requiring point-level accuracy, geometry-aware verification, and robustness to dependency-driven error propagation. To benchmark it, we introduce PAGE Bench, with 4,906 problems and over 224K process-supervised, pixel-level GUI actions. We further propose PAGER, a topology-aware agent that decomposes construction into dependency-structured planning and pixel-level execution. Pixel-grounded supervised tuning establishes executable action grammar, while precision-aligned reinforcement learning mitigates rollout-induced exposure bias through state-conditioned geometric feedback. Experiments reveal a pronounced Semantic-Execution Gap: general multimodal models can exceed 88% action type accuracy yet remain below 6% task success. PAGER closes this gap, delivering 4.1x higher task success than the strongest evaluated general baseline and raising step success rate from below 9% for GUI-specialized agents to over 62%, establishing a new state of the art for point-precise GUI control.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 58 references that are in the map. First posted on arXiv 15 May 2026.

Builds on

44 of 58 references.

Closest ancestors · 18

Same direction, newest first

  1. 2026MedSPOT: A Workflow-Aware Sequential Grounding Benchmark for Clinical GUI20 Mar
  2. 2026Milestones over Outcome: Unlocking Geometric Reasoning with Sub-Goal Verifiable Reward8 Jan
  3. 2025History-Aware Reasoning for GUI Agents12 Nov
  4. 2025GeoFM: Enhancing Geometric Reasoning of MLLMs via Synthetic Data Generation through Formal Language31 Oct
  5. 2025GeoVLMath: Enhancing Geometry Reasoning in Vision-Language Models via Cross-Modal Reward for Auxiliary Line Creation13 Oct
  6. 2025GeoSketch: A Neural-Symbolic Approach to Geometric Multimodal Reasoning with Auxiliary Line Construction and Affine Transformation26 Sep
  7. 2025OpenCUA: Open Foundations for Computer-Use Agents12 Aug
  8. 2025UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Grounding29 Jul
  9. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning1 Jul
  10. 2025ReGUIDE: Data Efficient GUI Grounding via Spatial Reasoning and Search21 May
  11. 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasoners19 Apr
  12. 2025GeoSense: Evaluating Identification and Application of Geometric Principles in Multimodal Reasoning17 Apr
6 more
  1. 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents14 Apr
  2. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization17 Mar
  3. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
  4. 2024GUICourse: From General Vision Language Models to Versatile GUI Agents17 Jun
  5. 2024CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation19 Feb
  6. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec

Other references · 26

  1. 2025Qwen3-VL Technical Reportcited by 2,793
  2. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  3. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  4. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 647
  5. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 610
  6. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 572
  7. 2025Kimi-VL Technical Reportcited by 388
  8. 2024GUI Agents: A Surveycited by 103
  9. 2025Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learningcited by 81
  10. 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agentscited by 74
  11. 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflectioncited by 67
  12. 2025MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environmentcited by 52
14 more
  1. 2025MobileRL: Online Agentic Reinforcement Learning for Mobile GUI Agentscited by 43
  2. 2025MCPToolBench++: A Large Scale AI Agent Model Context Protocol MCP Tool Use Benchmarkcited by 32
  3. 2025MM-PRM: Enhancing Multimodal Mathematical Reasoning with Scalable Step-Level Supervisioncited by 31
  4. 2025WorldGUI: An Interactive Benchmark for Desktop GUI Automation from Any Starting Pointcited by 20
  5. 2025SOLIDGEO: Measuring Multimodal Spatial Math Reasoning in Solid Geometrycited by 18
  6. 2026GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RLcited by 16
  7. 2025ProBench: Benchmarking GUI Agents with Accurate Process Informationcited by 15
  8. 2025MATP-BENCH: Can MLLM Be a Good Automated Theorem Prover for Multimodal Problems?cited by 13
  9. 2025Geoint-R1: Formalizing Multimodal Geometric Reasoning with Dynamic Auxiliary Constructionscited by 11
  10. 2025GUI-360^circ: A Comprehensive Dataset and Benchmark for Computer-Using Agentscited by 10
  11. 2024VisAidMath: Benchmarking Visual-Aided Mathematical Reasoningcited by 9
  12. 2025GeoBench: Rethinking Multimodal Geometric Problem-Solving via Hierarchical Evaluationcited by 4
  13. 2025Beyond Clicking:A Step Towards Generalist GUI Grounding via Text Draggingcited by 3
  14. 2026GeoChallenge: A Multi-Answer Multiple-Choice Benchmark for Geometric Reasoning with Diagramscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2026GUI-Primitives: Diagnosing Spatial Reasoning Failures in Vision-Language GUI Grounding
  2. 2026HATS: Hardness-Aware Trajectory Synthesis for GUI Agents
  3. 2026Learn from Move: the Next Step for GUI Agents
  4. 2024Ponder & Press: Advancing Visual GUI Agent towards General Computer Control
  5. 2025Breaking the Data Barrier -- Building GUI Agents Through Task Generalization
  6. 2026Measure Twice, Click Once: Co-evolving Proposer and Visual Critic via Reinforcement Learning for GUI Grounding
  7. 2026ActionEngine: From Reactive to Programmatic GUI Agents via State Machine Memory
  8. 2026GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents