mlsanityData to 14 Oct 2026

Measure Twice, Click Once: Co-evolving Proposer and Visual Critic via Reinforcement Learning for GUI Grounding

arXiv 2604.21268 ↗First posted 23 Apr 2026cs.LG cs.AI cs.CVcited by 5

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Graphical User Interface (GUI) grounding requires mapping natural language instructions to precise pixel coordinates. However, due to visually homogeneous elements and dense layouts, models typically grasp semantic intent yet struggle with achieving precise localization. While scaling sampling attempts (Pass@k) reveals potential gains, static self-consistency strategies derived from geometric clustering often yield limited improvements, as the model's predictions tend to be spatially dispersed. In this paper, we propose replacing static consistency strategies with a learnable selection mechanism that selects the optimal target by critiquing its own proposals rendered on the screenshot. Given the significant disparity between the model's grounding and critiquing capabilities, we propose a co-evolving Propose-then-Critic framework. To jointly optimize these, we introduce a maturity-aware adaptive co-evolutionary reinforcement learning paradigm. This approach dynamically balances the training objectives of proposer and critic, where the diversity of the proposer's outputs enhances critic robustness, while the critic's maturing discrimination capability conversely unlocks the proposer's potential for extensive spatial exploration, fostering the mutual reinforcement and co-evolution of both capabilities, thereby ensuring generalizability to adapt to diverse and complex interface layouts. Extensive experiments over 6 benchmarks show that our method significantly enhances both grounding accuracy and critic reliability.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 29 of its 43 references that are in the map. First posted on arXiv 23 Apr 2026.

Builds on

29 of 43 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Test-Time Reinforcement Learning for GUI Grounding via Region Consistency7 Aug
  2. 2025Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning18 May
  3. 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learning27 Mar
  4. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training28 Jan

Other references · 25

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 665
  3. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 567
  4. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 382
  5. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 378
  6. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 301
  7. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 278
  8. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 254
  9. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 181
  10. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 157
  11. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 139
  12. 2025GTA1: GUI Test-time Scaling Agentcited by 106
13 more
  1. 2020Widget Captioning: Generating Natural Language Description for Mobile User Interface Elementscited by 95
  2. 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 77
  3. 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflectioncited by 67
  4. 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 67
  5. 2025Guided Self-Evolving LLMs with Minimal Human Supervisioncited by 35
  6. 2025DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoningcited by 31
  7. 2025Visual Test-time Scaling for GUI Agent Groundingcited by 30
  8. 2025UI-E2I-Synth: Advancing GUI Grounding with Large-Scale Instruction Synthesiscited by 20
  9. 2024Improved GUI Grounding via Iterative Narrowingcited by 15
  10. 2024Attention-driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models without Fine-Tuningcited by 14
  11. 2026Verified Critical Step Optimization for LLM Agentscited by 14
  12. 2025Learning GUI Grounding with Spatial Reasoning from Visual Feedbackcited by 12
  13. 2025textscGUI-Spotlight: Adaptive Iterative Focus Refinement for Enhanced GUI Visual Groundingcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents
  2. 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents
  3. 2025GuirlVG: Incentivize GUI Visual Grounding via Empirical Exploration on Reinforcement Learning
  4. 2026AutoFocus: Uncertainty-Aware Active Visual Search for GUI Grounding
  5. 2026Test-Time Self-Evolving GUI Visual Grounding via Reflection-Guided On-Policy Self-Distillation
  6. 2025UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Grounding
  7. 2025GUI-ARP: Enhancing Grounding with Adaptive Region Perception for GUI Agents
  8. 2025MEGA-GUI: Multi-stage Enhanced Grounding Agents for GUI Elements