mlsanityData to 14 Oct 2026

GUI-G2: Gaussian Reward Modeling for GUI Grounding

arXiv 2507.15846 ↗First posted 21 Jul 2025cs.LG cs.AI cs.CLcited by 63

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

Graphical User Interface (GUI) grounding maps natural language instructions to precise interface locations for autonomous interaction. Current reinforcement learning approaches use binary rewards that treat elements as hit-or-miss targets, creating sparse signals that ignore the continuous nature of spatial interactions. Motivated by human clicking behavior that naturally forms Gaussian distributions centered on target elements, we introduce GUI Gaussian Grounding Rewards (GUI-G2), a principled reward framework that models GUI elements as continuous Gaussian distributions across the interface plane. GUI-G2 incorporates two synergistic mechanisms: Gaussian point rewards model precise localization through exponentially decaying distributions centered on element centroids, while coverage rewards assess spatial alignment by measuring the overlap between predicted Gaussian distributions and target regions. To handle diverse element scales, we develop an adaptive variance mechanism that calibrates reward distributions based on element dimensions. This framework transforms GUI grounding from sparse binary classification to dense continuous optimization, where Gaussian distributions generate rich gradient signals that guide models toward optimal interaction positions. Extensive experiments across ScreenSpot, ScreenSpot-v2, and ScreenSpot-Pro benchmarks demonstrate that GUI-G2, substantially outperforms state-of-the-art method UI-TARS-72B, with the most significant improvement of 24.7% on ScreenSpot-Pro. Our analysis reveals that continuous modeling provides superior robustness to interface variations and enhanced generalization to unseen layouts, establishing a new paradigm for spatial reasoning in GUI interaction tasks.

Key facts. Cited by 63 papers in the map (data through 14 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 36 of its 44 references that are in the map. First posted on arXiv 21 Jul 2025.

Builds on

36 of 44 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025LPO: Towards Accurate GUI Agent Interaction via Location Preference Optimization11 Jun

Other references · 35

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  4. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  5. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
  6. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 567
  7. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 526
  8. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 484
  9. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 382
  10. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 378
  11. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 319
  12. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 293
23 more
  1. 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agentscited by 244
  2. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 232
  3. 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 209
  4. 2024OmniParser for Pure Vision Based GUI Agentcited by 198
  5. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaborationcited by 189
  6. 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learningcited by 168
  7. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesiscited by 163
  8. 2024Large Language Model-Brained GUI Agents: A Surveycited by 162
  9. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 157
  10. 2020PP-OCR: A Practical Ultra Lightweight OCR Systemcited by 151
  11. 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasonerscited by 140
  12. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 139
  13. 2025Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Taskscited by 131
  14. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 95
  15. 2025Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learningcited by 81
  16. 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agentscited by 74
  17. 2025A Survey on (M)LLM-Based GUI Agentscited by 62
  18. 2025AppAgentX: Evolving GUI Agents as Proficient Smartphone Userscited by 46
  19. 2025UFO2: The Desktop AgentOScited by 41
  20. 2025Video Compression Commander: Plug-and-Play Inference Acceleration for Video Large Language Modelscited by 38
  21. 2025Global Compression Commander: Plug-and-Play Inference Acceleration for High-Resolution Large Vision-Language Modelscited by 32
  22. 2025Shifting AI Efficiency From Model-Centric to Data-Centric Compressioncited by 29
  23. 2025Think Twice, Click Once: Enhancing GUI Grounding via Fast and Slow Systemscited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 24; 2026: 39.

63 papers.

Sort:

By area:

  1. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 190
  2. 2026Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agentscited by 79
  3. 2025UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learningcited by 32
  4. 2025InfiGUI-G1: Advancing GUI Grounding with Adaptive Exploration Policy Optimizationcited by 27
  5. 2025UItron: Foundational GUI Agent with Advanced Perception and Planningcited by 22
  6. 2025Grounding Computer Use Agents on Human Demonstrationscited by 22
  7. 2026KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluationcited by 21
  8. 2025Ferret-UI Lite: Lessons from Building Small On-Device GUI Agentscited by 16
  9. 2026GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RLcited by 16
  10. 2025VeriOS: Query-Driven Proactive Human-Agent-GUI Interaction for Trustworthy OS Agentscited by 15
  11. 2025Mixing Importance with Diversity: Joint Optimization for KV Cache Compression in Large Vision-Language Modelscited by 14
  12. 2026ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agentscited by 14
38 more
  1. 2025GUI-ReWalk: Massive Data Generation for GUI Agent via Stochastic Exploration and Intent-Aware Reasoningcited by 13
  2. 2026OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Executioncited by 13
  3. 2025IAG: Input-aware Backdoor Attack on VLM-based Visual Groundingcited by 12
  4. 2025Learning GUI Grounding with Spatial Reasoning from Visual Feedbackcited by 12
  5. 2025GhostEI-Bench: Do Mobile Agents Resilience to Environmental Injection in Dynamic On-Device Environments?cited by 12
  6. 2025Quick on the Uptake: Eliciting Implicit Intents from Human Demonstrations for Personalized Mobile-Use Agentscited by 11
  7. 2025ProRe: A Proactive Reward System for GUI Agents via Reasoner-Actor Collaborationcited by 11
  8. 2026Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Groundingcited by 11
  9. 2025GUI-ARP: Enhancing Grounding with Adaptive Region Perception for GUI Agentscited by 10
  10. 2025GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Groundingcited by 8
  11. 2025VenusBench-GD: A Comprehensive Multi-Platform GUI Benchmark for Diverse Grounding Taskscited by 7
  12. 2026Unified Generation and Self-Verification for Vision-Language Models via Advantage Decoupled Preference Optimizationcited by 7
  13. 2026Continual GUI Agentscited by 7
  14. 2026AdaZoom-GUI: Adaptive Zoom-based GUI Grounding with Instruction Refinementcited by 6
  15. 2026Zoom to Essence: Trainless GUI Grounding by Inferring upon Interface Elementscited by 5
  16. 2025SWIRL: A Staged Workflow for Interleaved Reinforcement Learning in Mobile GUI Controlcited by 4
  17. 2025Generalist Scanner Meets Specialist Locator: A Synergistic Coarse-to-Fine Framework for Robust GUI Groundingcited by 4
  18. 2026What Happens Before Decoding? Prefill Determines GUI Grounding in VLMscited by 4
  19. 2025Poivre: Self-Refining Visual Pointing with Reinforcement Learningcited by 3
  20. 2025Beyond Clicking:A Step Towards Generalist GUI Grounding via Text Draggingcited by 3
  21. 2026CGL: Advancing Continual GUI Learning via Reinforcement Fine-Tuningcited by 3
  22. 2026GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agentscited by 3
  23. 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitantscited by 3
  24. 2026BAMI: Training-Free Bias Mitigation in GUI Groundingcited by 3
  25. 2025InfraMind: A Novel Exploration-based GUI Agentic Framework for Mission-critical Industrial Managementcited by 2
  26. 2025RISK: A Framework for GUI Agents in E-commerce Risk Managementcited by 2
  27. 2026One Forward Beats Two: InnerZoom for Accurate and Efficient GUI Groundingcited by 2
  28. 2026Reflect, Revise, Reuse: Training-Free Skill Evolution for GUI Agentscited by 2
  29. 2026SCA: Spatial Credit Assignment for Reinforcement Learning of GUI Agentscited by 2
  30. 2025Training High-Level Schedulers with Execution-Feedback Reinforcement Learning for Long-Horizon GUI Automationcited by 1
  31. 2026Building Autonomous GUI Navigation via Agentic-Q Estimation and Step-Wise Policy Optimizationcited by 1
  32. 2026UIPress: Bringing Optical Token Compression to UI-to-Code Generationcited by 1
  33. 2026Faithful Mobile GUI Agents with Guided Advantage Estimatorcited by 1
  34. 2026Scaling, Benchmarking, and Reasoning of Vision-Language Agents for Mobile GUI Navigationcited by 1
  35. 2026GUI-AC: Enhancing Continual Learning in GUI Agentscited by 1
  36. 2026SwipeGen: Bridging the Execution Gap in GUI Agents via Human-like Swipe Synthesiscited by 0
  37. 2026SSL: Sweet Spot Learning for Differentiated Guidance in Agentic Optimizationcited by 0
  38. 2026Trifuse: Enhancing Attention-Based GUI Grounding via Multimodal Fusioncited by 0

50 most cited of 63 All 63 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Measure Twice, Click Once: Co-evolving Proposer and Visual Critic via Reinforcement Learning for GUI Grounding
  2. 2026GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents
  3. 2025GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration
  4. 2025Test-Time Reinforcement Learning for GUI Grounding via Region Consistency
  5. 2025GuirlVG: Incentivize GUI Visual Grounding via Empirical Exploration on Reinforcement Learning
  6. 2025Enhancing Trustworthy GUI Grounding via Self-Critiqued Reinforcement Learning
  7. 2026MagicGUI-RMS: A Multi-Agent Reward Model System for Self-Evolving GUI Agents via Automated Feedback Reflux
  8. 2026Seg2Click: Learning Pixel-Level Action Masks for GUI Visual Grounding