GuirlVG: Incentivize GUI Visual Grounding via Empirical Exploration on Reinforcement Learning
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
also in 2 more
Abstract (arXiv)
Graphical user interface visual grounding (GUI-VG), a core capability for GUI agents, has primarily relied on supervised fine-tuning (SFT) of multimodal large language models (MLLMs), which demands extensive data curation and significant training costs. However, as MLLMs continue to advance and even cover GUI domains during pretraining, the necessity of exhaustive SFT post-training becomes increasingly questionable. Meanwhile, recent successes of rule-based reinforcement fine-tuning (RFT) suggest a more efficient alternative. Despite this promise, the optimal manner of applying RFT for GUI-VG remains unexplored. To bridge this gap, we introduce GuirlVG, a reinforcement learning-based GUI-VG method built on a systematic empirical study and a novel stabilization technique. We find that naive application of RFT underperforms the SFT baseline, motivating a deeper exploration. First, we decompose RFT into its core components and analyze the optimal formulation of each. Second, we propose a novel Adversarial KL Factor that dynamically stabilizes training to mitigate reward over-optimization. Third, we further explore the training configurations of RFT to enhance effectiveness. Extensive experiments show that GuirlVG, with only 5.2K training samples, outperforms SFT methods trained on over 10M samples, achieving a 7.7% improvement on ScreenSpot, a 17.2% improvement on ScreenSpotPro, and 91.9% accuracy on ScreenSpotV2.
Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 46 references that are in the map. First posted on arXiv 6 Aug 2025.
Builds on
38 of 46 references.
Closest ancestors · 8
Same direction, newest first
- 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model10 Apr
- 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use4 Apr
- 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RL10 Mar
- 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models9 Mar
- 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
- 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
- 2024OmniParser for Pure Vision Based GUI Agent1 Aug
- 2023CogAgent: A Visual Language Model for GUI Agents14 Dec
Other references · 30
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2024GPT-4o System Cardcited by 4,588
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
- 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
- 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
18 more
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 610
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 567
- 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 513
- 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 278
- 2021TransVG: End-to-End Visual Grounding with Transformerscited by 271
- 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
- 2025Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcementcited by 239
- 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 232
- 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 157
- 2024Aria-UI: Visual Grounding for GUI Instructionscited by 140
- 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 139
- 2024Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?cited by 56
- 2024Robin3D: Improving 3D Large Language Model via Robust Instruction Tuningcited by 26
- 2025SpiritSight Agent: Advanced GUI Agent with One Lookcited by 18
- 2024Intent3D: 3D Object Detection in RGB-D Scans Based on Human Intentioncited by 16
- 2024SegVG: Transferring Object Bounding Box to Segmentation for Visual Groundingcited by 14
- 2024Visual Grounding with Attention-Driven Constraint Balancingcited by 7
- 2024ACTRESS: Active Retraining for Semi-supervised Visual Groundingcited by 7
Built on by
Citing papers per year: 2025: 2; 2026: 5.
7 papers.
- 2026Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Groundingcited by 11
- 2025textscGUI-Spotlight: Adaptive Iterative Focus Refinement for Enhanced GUI Visual Groundingcited by 10
- 2025VGent: Visual Grounding via Modular Design for Disentangling Reasoning and Predictioncited by 7
- 2026What Happens Before Decoding? Prefill Determines GUI Grounding in VLMscited by 4
- 2026GUI-Perturbed: Domain Randomization Reveals Systematic Brittleness in GUI Grounding Modelscited by 3
- 2026One Forward Beats Two: InnerZoom for Accurate and Efficient GUI Groundingcited by 2
- 2026Learning from Reliable Negatives: Confidence-Anchored Test-Time Adaptation for GUI Groundingcited by 0
- 2026Learning from Reliable Negatives: Confidence-Anchored Test-Time Adaptation for GUI Grounding14 Sep 2026
- 2026One Forward Beats Two: InnerZoom for Accurate and Efficient GUI Grounding29 Jun 2026
- 2026What Happens Before Decoding? Prefill Determines GUI Grounding in VLMs10 May 2026
- 2026Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding1 May 2026
- 2026GUI-Perturbed: Domain Randomization Reveals Systematic Brittleness in GUI Grounding Models15 Apr 2026
- 2025VGent: Visual Grounding via Modular Design for Disentangling Reasoning and Prediction11 Dec 2025
- 2025textscGUI-Spotlight: Adaptive Iterative Focus Refinement for Enhanced GUI Visual Grounding5 Oct 2025
Similar, not cited
Close in wording, no citation either way
- 2025Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning
- 2026GroundGUI: GUI Grounding by High-Quality Data and Distillation-to-Reinforcement Learning
- 2026GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents
- 2025GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration
- 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents
- 2025UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Grounding
- 2026Measure Twice, Click Once: Co-evolving Proposer and Visual Critic via Reinforcement Learning for GUI Grounding
- 2025Learning GUI Grounding with Spatial Reasoning from Visual Feedback