mlsanityData to 14 Oct 2026

Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning

arXiv 2505.12370 ↗First posted 18 May 2025cs.AIcited by 81

On the map: LLM agents and tool use → GUI and mobile agents

also in 1 more
Abstract (arXiv)

Graphical User Interface (GUI) agents have made substantial strides in understanding and executing user instructions across diverse platforms. Yet, grounding these instructions to precise interface elements remains challenging, especially in complex, high-resolution, professional environments. Traditional supervised finetuning (SFT) methods often require large volumes of diverse data and exhibit weak generalization. To overcome these limitations, we introduce a reinforcement learning (RL) based framework that incorporates three core strategies: (1) seed data curation to ensure high quality training samples, (2) a dense policy gradient that provides continuous feedback based on prediction accuracy, and (3) a self evolutionary reinforcement finetuning mechanism that iteratively refines the model using attention maps. With only 3k training samples, our 7B-parameter model achieves state-of-the-art results among similarly sized models on three grounding benchmarks. Notably, it attains 47.3% accuracy on the ScreenSpot-Pro dataset, outperforming much larger models, such as UI-TARS-72B, by a margin of 24.2%. These findings underscore the effectiveness of RL-based approaches in enhancing GUI agent performance, particularly in high-resolution, complex environments.

Key facts. Cited by 81 papers in the map (data through 14 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 28 of its 38 references that are in the map. First posted on arXiv 18 May 2025.

Builds on

28 of 38 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents14 Apr
  2. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use4 Apr
  3. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
  4. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
  5. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
  6. 2024AppAgent v2: Advanced Agent for Flexible Mobile Interactions5 Aug
  7. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration3 Jun
  8. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web27 Feb
  9. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception29 Jan
  10. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan
  11. 2023AppAgent: Multimodal Agents as Smartphone Users21 Dec
  12. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec

Other references · 16

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2025Qwen2.5-VL Technical Reportcited by 5,587
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2024GPT-4o System Cardcited by 4,588
  7. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 665
  8. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 526
  9. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 513
  10. 2025LIMO: Less is More for Reasoningcited by 498
  11. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 278
  12. 2023AutoDroid: LLM-powered Task Automation in Androidcited by 184
4 more
  1. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 140
  2. 2025LIMR: Less is More for RL Scalingcited by 139
  3. 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 77
  4. 2025100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Modelscited by 30

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 35; 2026: 46.

81 papers.

Sort:

By area:

  1. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 190
  2. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 170
  3. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  4. 2026Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forwardcited by 113
  5. 2025GTA1: GUI Test-time Scaling Agentcited by 106
  6. 2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 93
  7. 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agentscited by 91
  8. 2025GUI-G2: Gaussian Reward Modeling for GUI Groundingcited by 63
  9. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 60
  10. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  11. 2026UI-Venus-1.5 Technical Reportcited by 41
  12. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 40
38 more
  1. 2025Ivy-Fake: A Unified Explainable Framework and Benchmark for Image and Video AIGC Detectioncited by 34
  2. 2025DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoningcited by 31
  3. 2025InfiGUI-G1: Advancing GUI Grounding with Adaptive Exploration Policy Optimizationcited by 27
  4. 2025UItron: Foundational GUI Agent with Advanced Perception and Planningcited by 22
  5. 2025UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoningcited by 22
  6. 2025Grounding Computer Use Agents on Human Demonstrationscited by 22
  7. 2025Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skillscited by 22
  8. 2025Ferret-UI Lite: Lessons from Building Small On-Device GUI Agentscited by 16
  9. 2026MolmoPoint: Better Pointing for VLMs with Grounding Tokenscited by 15
  10. 2026OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Executioncited by 13
  11. 2025Learning GUI Grounding with Spatial Reasoning from Visual Feedbackcited by 12
  12. 2025MVP: Multiple View Prediction Improves GUI Groundingcited by 12
  13. 2026Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Groundingcited by 11
  14. 2025GUI-ARP: Enhancing Grounding with Adaptive Region Perception for GUI Agentscited by 10
  15. 2025textscGUI-Spotlight: Adaptive Iterative Focus Refinement for Enhanced GUI Visual Groundingcited by 10
  16. 2026GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agentscited by 10
  17. 2025Reinforcement Learning for Large Model: A Surveycited by 9
  18. 2025Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AIcited by 8
  19. 2025GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Groundingcited by 8
  20. 2026UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Groundingcited by 8
  21. 2025Chain-of-Ground: Improving GUI Grounding via Iterative Reasoning and Reference Feedbackcited by 7
  22. 2025VenusBench-GD: A Comprehensive Multi-Platform GUI Benchmark for Diverse Grounding Taskscited by 7
  23. 2026Continual GUI Agentscited by 7
  24. 2025TabDSR: Decompose, Sanitize, and Reason for Complex Numerical Reasoning in Tabular Datacited by 5
  25. 2025The Consistency Critic: Correcting Inconsistencies in Generated Images via Reference-Guided Attentive Alignmentcited by 5
  26. 2026Zoom to Essence: Trainless GUI Grounding by Inferring upon Interface Elementscited by 5
  27. 2026Measure Twice, Click Once: Co-evolving Proposer and Visual Critic via Reinforcement Learning for GUI Groundingcited by 5
  28. 2026AEM: Adaptive Entropy Modulation for Multi-Turn Agentic Reinforcement Learningcited by 5
  29. 2026DRS-GUI: Dynamic Region Search for Training-Free GUI Groundingcited by 5
  30. 2026Learning from Failure: Inference-Time Self-Improvement for Computer-Use Agentscited by 5
  31. 2025SWIRL: A Staged Workflow for Interleaved Reinforcement Learning in Mobile GUI Controlcited by 4
  32. 2025Improving GUI Grounding with Explicit Position-to-Coordinate Mappingcited by 4
  33. 2026POINTS-GUI-G: GUI-Grounding Journeycited by 4
  34. 2026Zoom Consistency: A Free Confidence Signal in Multi-Step Visual Grounding Pipelinescited by 4
  35. 2026LiteGUI: Distilling Compact GUI Agents with Reinforcement Learningcited by 4
  36. 2026What Happens Before Decoding? Prefill Determines GUI Grounding in VLMscited by 4
  37. 2026SE-GA: Memory-Augmented Self-Evolution for GUI Agentscited by 4
  38. 2025Learning Active Perception via Self-Evolving Preference Optimization for GUI Groundingcited by 3

50 most cited of 81 All 81 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025GuirlVG: Incentivize GUI Visual Grounding via Empirical Exploration on Reinforcement Learning
  2. 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents
  3. 2025GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration
  4. 2025UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Grounding
  5. 2025GUI-Shift: Enhancing VLM-Based GUI Agents through Self-supervised Reinforcement Learning
  6. 2025Enhancing Trustworthy GUI Grounding via Self-Critiqued Reinforcement Learning
  7. 2025GUI Exploration Lab: Enhancing Screen Navigation in Agents via Multi-Turn Reinforcement Learning
  8. 2025CRAFT-GUI: Curriculum-Reinforced Agent For GUI Tasks