GUI-C2: Coarse-to-Fine GUI Grounding via Difficulty-Aware Reinforcement Learning
On the map: LLM agents and tool use → GUI and mobile agents
Abstract (arXiv)
Existing agentic reinforcement learning methods for GUI grounding have limitations at two levels. At the data level, current approaches typically treat all training samples equally, although their training value to the baseline model varies with difficulty. Overlooking this can greatly reduce training efficiency or even cause collapse. At the strategy level, existing frameworks struggle to balance the trade-off between cropping larger regions for sufficient context and smaller ones for reduced redundancy, a tension inherent to tool-augmented grounding agents. In addition, overly complex decision-making is difficult for small-parameter models and significantly increases inference time. To address these issues, at the data level, we propose GUI-D, a data mining and difficulty scoring pipeline that identifies the training-worthy samples by proper testing and assigns difficulty scores to guide subsequent training weights. At the strategy level, we propose GUI-C2, which employs an area-gated coarse-to-fine refinement mechanism that progressively narrows the visual field via model-internal uncertainty signals, adaptively reserving context for large targets while amplifying precision for small ones, reinforced by improvement-aware stage rewards that ensure each refinement genuinely advances grounding. Meanwhile, we simplify the decision-making process to greatly reduce additional inference time. Finally, extensive experiments show that our method achieves state-of-the-art performance. The code and data will be publicly available.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 28 of its 39 references that are in the map. First posted on arXiv 29 May 2026.
Builds on
28 of 39 references.
Closest ancestors · 11
Same direction, newest first
- 2025UI-Venus Technical Report: Building High-performance UI Agents with RFT14 Aug
- 2025OpenCUA: Open Foundations for Computer-Use Agents12 Aug
- 2025Uncertainty-Aware GUI Agent: Adaptive Perception through Component Recommendation and Human-in-the-Loop Refinement6 Aug
- 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents3 Jun
- 2025Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning18 May
- 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents14 Apr
- 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use4 Apr
- 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learning27 Mar
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan
- 2023CogAgent: A Visual Language Model for GUI Agents14 Dec
Other references · 17
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
- 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 526
- 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
- 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 232
- 2025GTA1: GUI Test-time Scaling Agentcited by 106
- 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 95
- 2025DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoningcited by 31
- 2025LangTime: A Language-Guided Unified Model for Time Series Forecasting with Proximal Policy Optimizationcited by 22
- 2026UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Groundingcited by 8
- 2026AdaZoom-GUI: Adaptive Zoom-based GUI Grounding with Instruction Refinementcited by 6
- 2025Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challengescited by 5
5 more
- 2026Zoom to Essence: Trainless GUI Grounding by Inferring upon Interface Elementscited by 5
- 2026GUI-Perturbed: Domain Randomization Reveals Systematic Brittleness in GUI Grounding Modelscited by 3
- 2026BAMI: Training-Free Bias Mitigation in GUI Groundingcited by 3
- 2025Dynamic Collaboration of Multi-Language Models based on Minimal Complete Semantic Unitscited by 2
- 2026Seg-Agent: Test-Time Multimodal Reasoning for Training-Free Language-Guided Segmentationcited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025GuirlVG: Incentivize GUI Visual Grounding via Empirical Exploration on Reinforcement Learning
- 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents
- 2025UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Grounding
- 2025GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration
- 2025Test-Time Reinforcement Learning for GUI Grounding via Region Consistency
- 2025CRAFT-GUI: Curriculum-Reinforced Agent For GUI Tasks
- 2026Continual GUI Agents
- 2026GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents