Trust the Right Teacher: Quality-Aware Self-Distillation for GUI Grounding
On the map: Knowledge distillation → Multimodal distillation
Abstract (arXiv)
Graphical user interface (GUI) grounding requires vision-language models (VLMs) to identify small target elements in high-resolution screenshots and predict precise screen coordinates. On-policy self-distillation (OPSD) is a promising post-training approach for this coordinate-sensitive task, since it provides dense token-level teacher signals beyond hard coordinate labels. However, naive OPSD is not well suited to GUI grounding: OPSD evaluates the teacher on student-generated prefixes, the quality of coordinate-token teacher signals can degrade when the prefix has already deviated from the target coordinate, leading to unreliable teacher signal. To mitigate this, We propose quality-aware self-distillation for VLM-based GUI grounding, which improves coordinate-token teacher-signal quality through soft correctness-aware gating and teacher-probability scaling. The soft correctness-aware gate checks whether the teacher's current coordinate-token prediction can still be completed into the ground-truth box under the student-generated prefix. If not, the corresponding teacher signal is down-weighted. Teacher-probability scaling then uses the teacher's confidence as a lightweight factor to further calibrate the strength of the gated supervision. A key empirical finding is that neither component alone improves overall performance, whereas combining them consistently improves performance. This suggests that the two mechanisms play complementary roles: correctness-aware gating suppresses unreliable coordinate-token supervision, while teacher-probability scaling calibrates the strength of the remaining signals. Experiments across six GUI grounding benchmarks show that our method consistently improves the base model and outperforms strong baselines.
Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Multimodal distillation, one of 12 directions in Knowledge distillation. Builds on 26 of its 30 references that are in the map. First posted on arXiv 16 Jun 2026.
Builds on
26 of 30 references.
References · 26
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
- 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
- 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
- 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 567
- 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 455
- 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 382
- 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 378
- 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 139
14 more
- 2022Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generationcited by 108
- 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 77
- 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agentscited by 74
- 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 67
- 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 60
- 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixescited by 45
- 2026SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weightingcited by 43
- 2025Towards Trustworthy GUI Agents: A Surveycited by 34
- 2026Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoningcited by 23
- 2025R-VLM: Region-Aware Vision Language Model for Precise GUI Groundingcited by 22
- 2025Learning GUI Grounding with Spatial Reasoning from Visual Feedbackcited by 12
- 2026Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Groundingcited by 11
- 2026PAINT: Partial-Solution Adaptive Interpolated Training for Self-Distilled Reasonerscited by 8
- 2026TRON: Targeted Rule-Verifiable Online Environments for Visual Reasoning RLcited by 3
Built on by
Citing papers per year: 2026: 4.
4 papers.
- 2026Self-Supervised Scaling of Terminal Environments for Scientific Domainscited by 1
- 2026Learn How to Act from Your Own Interactions: On-Policy Self-Distillation for GUI Agentscited by 0
- 2026Train4Merge: A Controlled Single-Teacher Study of RL vs. SFT Teachers for OPD-Based Model Mergingcited by 0
- 2026Recurrent Latent Visual Search for GUI Groundingcited by 0
- 2026Recurrent Latent Visual Search for GUI Grounding4 Oct 2026
- 2026Self-Supervised Scaling of Terminal Environments for Scientific Domains2 Oct 2026
- 2026Train4Merge: A Controlled Single-Teacher Study of RL vs. SFT Teachers for OPD-Based Model Merging26 Sep 2026
- 2026Learn How to Act from Your Own Interactions: On-Policy Self-Distillation for GUI Agents23 Sep 2026
Similar, not cited
Close in wording, no citation either way
- 2026LookAgain: Closed-Loop GUI Grounding with Visually Grounded Reflection
- 2024Improved GUI Grounding via Iterative Narrowing
- 2026FocusUI: Efficient UI Grounding via Position-Preserving Visual Token Selection
- 2026Hallucination-Free GUI Grounding via Regression-Free Layout-Aware Matching
- 2026GUI-Perturbed: Domain Randomization Reveals Systematic Brittleness in GUI Grounding Models
- 2026GUI-Lens: Coarse-to-Fine Cropping for GUI Grounding with General-Purpose VLMs
- 2026Seg2Click: Learning Pixel-Level Action Masks for GUI Visual Grounding
- 2025Chain-of-Ground: Improving GUI Grounding via Iterative Reasoning and Reference Feedback