# Measure Twice, Click Once: Co-evolving Proposer and Visual Critic via Reinforcement Learning for GUI Grounding

arXiv: https://arxiv.org/abs/2604.21268 · HTML: https://mlsanity.com/paper/2604.21268/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 29 of its 43 references that are in the map.
- First posted on arXiv 23 Apr 2026.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 29 of 29)

- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training](https://mlsanity.com/paper/2501.17161/) (2025)
- [Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models](https://mlsanity.com/paper/2503.06749/) (2025)
- [SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents](https://mlsanity.com/paper/2401.10935/) (2024)
- [Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents](https://mlsanity.com/paper/2410.05243/) (2024)
- [OS-ATLAS: A Foundation Action Model for Generalist GUI Agents](https://mlsanity.com/paper/2410.23218/) (2024)
- [R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization](https://mlsanity.com/paper/2503.12937/) (2025)
- [Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction](https://mlsanity.com/paper/2412.04454/) (2024)
- [ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use](https://mlsanity.com/paper/2504.07981/) (2025)
- [Mapping Natural Language Instructions to Mobile UI Action Sequences](https://mlsanity.com/paper/2005.03776/) (2020)
- [UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learning](https://mlsanity.com/paper/2503.21620/) (2025)
- [OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web](https://mlsanity.com/paper/2402.17553/) (2024)
- [Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis](https://mlsanity.com/paper/2505.13227/) (2025)
- [GTA1: GUI Test-time Scaling Agent](https://mlsanity.com/paper/2507.05791/) (2025)
- [Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements](https://mlsanity.com/paper/2010.04295/) (2020)
- [Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning](https://mlsanity.com/paper/2505.12370/) (2025)
- [UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction](https://mlsanity.com/paper/2503.15661/) (2025)
- [InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection](https://mlsanity.com/paper/2501.04575/) (2025)
- [MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents](https://mlsanity.com/paper/2507.19478/) (2025)
- [Guided Self-Evolving LLMs with Minimal Human Supervision](https://mlsanity.com/paper/2512.02472/) (2025)
- [DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoning](https://mlsanity.com/paper/2507.00008/) (2025)
- [Visual Test-time Scaling for GUI Agent Grounding](https://mlsanity.com/paper/2505.00684/) (2025)
- [Test-Time Reinforcement Learning for GUI Grounding via Region Consistency](https://mlsanity.com/paper/2508.05615/) (2025)
- [UI-E2I-Synth: Advancing GUI Grounding with Large-Scale Instruction Synthesis](https://mlsanity.com/paper/2504.11257/) (2025)
- [Improved GUI Grounding via Iterative Narrowing](https://mlsanity.com/paper/2411.13591/) (2024)
- [Attention-driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models without Fine-Tuning](https://mlsanity.com/paper/2412.10840/) (2024)
- [Verified Critical Step Optimization for LLM Agents](https://mlsanity.com/paper/2602.03412/) (2026)
- [Learning GUI Grounding with Spatial Reasoning from Visual Feedback](https://mlsanity.com/paper/2509.21552/) (2025)
- [textscGUI-Spotlight: Adaptive Iterative Focus Refinement for Enhanced GUI Visual Grounding](https://mlsanity.com/paper/2510.04039/) (2025)

## Built on by (most cited first, 5 of 5)

- [Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding](https://mlsanity.com/paper/2605.00642/) (2026)
- [LookAgain: Closed-Loop GUI Grounding with Visually Grounded Reflection](https://mlsanity.com/paper/2608.09723/) (2026)
- [Recurrent Latent Visual Search for GUI Grounding](https://mlsanity.com/paper/2610.05185/) (2026)
- [ConClick: Element-Wise Contrastive Learning Improves GUI Grounding](https://mlsanity.com/paper/2610.13268/) (2026)
- [GroundGUI: GUI Grounding by High-Quality Data and Distillation-to-Reinforcement Learning](https://mlsanity.com/paper/2610.19100/) (2026)

## Abstract (arXiv)

Graphical User Interface (GUI) grounding requires mapping natural language instructions to precise pixel coordinates. However, due to visually homogeneous elements and dense layouts, models typically grasp semantic intent yet struggle with achieving precise localization. While scaling sampling attempts (Pass@k) reveals potential gains, static self-consistency strategies derived from geometric clustering often yield limited improvements, as the model's predictions tend to be spatially dispersed. In this paper, we propose replacing static consistency strategies with a learnable selection mechanism that selects the optimal target by critiquing its own proposals rendered on the screenshot. Given the significant disparity between the model's grounding and critiquing capabilities, we propose a co-evolving Propose-then-Critic framework. To jointly optimize these, we introduce a maturity-aware adaptive co-evolutionary reinforcement learning paradigm. This approach dynamically balances the training objectives of proposer and critic, where the diversity of the proposer's outputs enhances critic robustness, while the critic's maturing discrimination capability conversely unlocks the proposer's potential for extensive spatial exploration, fostering the mutual reinforcement and co-evolution of both capabilities, thereby ensuring generalizability to adapt to diverse and complex interface layouts. Extensive experiments over 6 benchmarks show that our method significantly enhances both grounding accuracy and critic reliability.