Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding
On the map: LLM agents and tool use → GUI and mobile agents
also in 1 more
Abstract (arXiv)
Graphical User Interface (GUI) grounding maps natural language instructions to the visual coordinates of target elements and serves as a core capability for autonomous GUI agents. Recent reinforcement learning methods (e.g., GRPO) have achieved strong performance, but they rely on expensive multiple rollouts and suffer from sparse signals on hard samples. These limitations make on-policy self-distillation (OPSD), which provides dense token-level supervision from a single rollout, a promising alternative. However, its applicability to GUI grounding remains unexplored. In this paper, we present GUI-SD, the first OPSD framework tailored for GUI grounding. First, it constructs a visually enriched privileged context for the teacher using a target bounding box and a Gaussian soft mask, providing informative guidance without leaking exact coordinates. Second, it employs entropy-guided distillation, which adaptively weights tokens based on digit significance and teacher confidence, concentrating optimization on the most impactful and reliable positions. Extensive experiments on six representative GUI grounding benchmarks show that GUI-SD consistently outperforms GRPO-based methods and naive OPSD in both accuracy and training efficiency. Code and training data are available at https://zhangyan-ucas.github.io/GUI-SD/.
Key facts. Cited by 10 papers in the map (data through 13 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 49 of its 52 references that are in the map. First posted on arXiv 1 May 2026.
Builds on
49 of 52 references.
Closest ancestors · 24
Same direction, newest first
- 2026UI-Venus-1.5 Technical Report9 Feb
- 2026GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents14 Jan
- 2025Co-EPG: A Framework for Co-Evolution of Planning and Grounding in Autonomous GUI Agents13 Nov
- 2025UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoning23 Oct
- 2025BTL-UI: Blink-Think-Link Reasoning Model for GUI Agent19 Sep
- 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data18 Sep
- 2025MobileRL: Online Agentic Reinforcement Learning for Mobile GUI Agents10 Sep
- 2025ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agents19 Aug
- 2025GuirlVG: Incentivize GUI Visual Grounding via Empirical Exploration on Reinforcement Learning6 Aug
- 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents25 Jul
- 2025LPO: Towards Accurate GUI Agent Interaction via Location Preference Optimization11 Jun
- 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents3 Jun
12 more
- 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents21 May
- 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis19 May
- 2025Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning18 May
- 2025MobileIPL: Enhancing Mobile Agents Thinking Process via Iterative Preference Learning18 May
- 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasoners19 Apr
- 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents14 Apr
- 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use4 Apr
- 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learning27 Mar
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
- 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
- 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan
Other references · 25
- 2015Distilling the Knowledge in a Neural Networkcited by 10,180
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,091
- 2025Qwen3-VL Technical Reportcited by 2,663
- 2026Reinforcement Learning via Self-Distillationcited by 310
- 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 280
- 2026Self-Distillation Enables Continual Learningcited by 234
- 2026A Survey of On-Policy Distillation for Large Language Modelscited by 155
- 2026Self-Distilled RLVRcited by 121
- 2025GTA1: GUI Test-time Scaling Agentcited by 102
- 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 70
- 2025GUI-G2: Gaussian Reward Modeling for GUI Groundingcited by 61
- 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 49
13 more
- 2026Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perceptioncited by 48
- 2026POPE: Learning to Reason on Hard Problems via Privileged On-Policy Explorationcited by 40
- 2026OPSDL: On-Policy Self-Distillation for Long-Context Language Modelscited by 30
- 2025ToDi: Token-wise Distillation via Fine-Grained Divergence Controlcited by 26
- 2025GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Explorationcited by 18
- 2026MolmoWeb: Open Visual Web Agent and Open Data for the Open Webcited by 18
- 2026Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Modelscited by 9
- 2023EA-KD: Entropy-based Adaptive Knowledge Distillationcited by 8
- 2026WebFactory: Automated Compression of Foundational Language Intelligence into Grounded Web Agentscited by 6
- 2026Zoom to Essence: Trainless GUI Grounding by Inferring upon Interface Elementscited by 4
- 2025Enhancing Trustworthy GUI Grounding via Self-Critiqued Reinforcement Learningcited by 3
- 2026Measure Twice, Click Once: Co-evolving Proposer and Visual Critic via Reinforcement Learning for GUI Groundingcited by 3
- 2026Trust the uncertain teacher: distilling dark knowledge via calibrated uncertaintycited by 2
Built on by
Citing papers per year: 2026: 10.
10 papers.
- 2026Latent On-Policy Self-Distillationcited by 7
- 2026Trust the Right Teacher: Quality-Aware Self-Distillation for GUI Groundingcited by 4
- 2026SkillLens: Visual Skill Cards for Retrieval-Augmented GUI Action Prediction and On-Policy Distillationcited by 3
- 2026The Next Screenshot Knows: Gated Hindsight Distillation for Mobile GUI Agentscited by 1
- 2026Test-Time Self-Evolving GUI Visual Grounding via Reflection-Guided On-Policy Self-Distillationcited by 1
- 2026IoU-PD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Modelscited by 0
- 2026Towards Omni-dimensional GUI Agent Navigation with Masked Trajectory Predictioncited by 0
- 2026Learn How to Act from Your Own Interactions: On-Policy Self-Distillation for GUI Agentscited by 0
- 2026Train4Merge: A Controlled Single-Teacher Study of RL vs. SFT Teachers for OPD-Based Model Mergingcited by 0
- 2026Recurrent Latent Visual Search for GUI Groundingcited by 0
- 2026Recurrent Latent Visual Search for GUI Grounding4 Oct 2026
- 2026Train4Merge: A Controlled Single-Teacher Study of RL vs. SFT Teachers for OPD-Based Model Merging26 Sep 2026
- 2026Learn How to Act from Your Own Interactions: On-Policy Self-Distillation for GUI Agents23 Sep 2026
- 2026Towards Omni-dimensional GUI Agent Navigation with Masked Trajectory Prediction22 Sep 2026
- 2026Latent On-Policy Self-Distillation13 Aug 2026
- 2026Test-Time Self-Evolving GUI Visual Grounding via Reflection-Guided On-Policy Self-Distillation11 Aug 2026
- 2026SkillLens: Visual Skill Cards for Retrieval-Augmented GUI Action Prediction and On-Policy Distillation11 Aug 2026
- 2026The Next Screenshot Knows: Gated Hindsight Distillation for Mobile GUI Agents6 Aug 2026
- 2026IoU-PD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Models17 Jul 2026
- 2026Trust the Right Teacher: Quality-Aware Self-Distillation for GUI Grounding16 Jun 2026
Similar, not cited
Close in wording, no citation either way
- 2026WinDOM: Self-Family Distillation for Small-Model GUI Grounding
- 2026Learning from Reliable Negatives: Confidence-Anchored Test-Time Adaptation for GUI Grounding
- 2025Learning GUI Grounding with Spatial Reasoning from Visual Feedback
- 2025Test-Time Reinforcement Learning for GUI Grounding via Region Consistency
- 2025GUI-Shift: Enhancing VLM-Based GUI Agents through Self-supervised Reinforcement Learning
- 2025UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Grounding
- 2025MEGA-GUI: Multi-stage Enhanced Grounding Agents for GUI Elements
- 2025Chain-of-Ground: Improving GUI Grounding via Iterative Reasoning and Reference Feedback