# Visual-Advantage On-Policy Distillation for Vision-Language Models

arXiv: https://arxiv.org/abs/2605.21924 · HTML: https://mlsanity.com/paper/2605.21924/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 32 of its 32 references that are in the map.
- First posted on arXiv 21 May 2026.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)
- Knowledge distillation / [Multimodal distillation](https://mlsanity.com/distill/multimodal-distillation/)

## Builds on (most cited first, 30 of 32)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [Are We on the Right Way for Evaluating Large Vision-Language Models?](https://mlsanity.com/paper/2403.20330/) (2024)
- [A Diagram Is Worth A Dozen Images](https://mlsanity.com/paper/1603.07396/) (2016)
- [Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding](https://mlsanity.com/paper/2311.16922/) (2023)
- [MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?](https://mlsanity.com/paper/2403.14624/) (2024)
- [HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models](https://mlsanity.com/paper/2310.14566/) (2023)
- [Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning](https://mlsanity.com/paper/2506.01939/) (2025)
- [OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models](https://mlsanity.com/paper/2305.07895/) (2023)
- [Policy Distillation](https://mlsanity.com/paper/1511.06295/) (2015)
- [Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning](https://mlsanity.com/paper/2105.04165/) (2021)
- [VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning](https://mlsanity.com/paper/2504.08837/) (2025)
- [We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?](https://mlsanity.com/paper/2407.01284/) (2024)
- [Multi-Modal Hallucination Control by Visual Information Grounding](https://mlsanity.com/paper/2403.14003/) (2024)
- [Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding](https://mlsanity.com/paper/2403.18715/) (2024)
- [Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation](https://mlsanity.com/paper/2602.12125/) (2026)
- [Entropy-Aware On-Policy Distillation of Language Models](https://mlsanity.com/paper/2603.07079/) (2026)
- [DistiLLM: Towards Streamlined Distillation for Large Language Models](https://mlsanity.com/paper/2402.03898/) (2024)
- [Perception-Aware Policy Optimization for Multimodal Reasoning](https://mlsanity.com/paper/2507.06448/) (2025)
- [Self-Rewarding Vision-Language Model via Reasoning Decomposition](https://mlsanity.com/paper/2508.19652/) (2025)
- [Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling](https://mlsanity.com/paper/2410.11325/) (2024)
- [NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation](https://mlsanity.com/paper/2504.13055/) (2025)
- [V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimization](https://mlsanity.com/paper/2411.02712/) (2024)
- [Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models](https://mlsanity.com/paper/2404.02657/) (2024)
- [Spotlight on Token Perception for Multimodal Reinforcement Learning](https://mlsanity.com/paper/2510.09285/) (2025)
- [Visual Description Grounding Reduces Hallucinations and Boosts Reasoning in LVLMs](https://mlsanity.com/paper/2405.15683/) (2024)

## Built on by (most cited first, 24 of 24)

- [DOPD: Dual On-policy Distillation](https://mlsanity.com/paper/2606.30626/) (2026)
- [Visual Contrastive Self-Distillation](https://mlsanity.com/paper/2607.21556/) (2026)
- [Weak-to-Strong On-Policy Distillation](https://mlsanity.com/paper/2607.26246/) (2026)
- [OPD-V: Visual On-Policy Self-Distillation with Modality Balance](https://mlsanity.com/paper/2608.05131/) (2026)
- [VAD: Attributing Visual Evidence for Target Reconstruction in Multimodal On-Policy Distillation](https://mlsanity.com/paper/2607.28590/) (2026)
- [Latent On-Policy Self-Distillation](https://mlsanity.com/paper/2608.13040/) (2026)
- [Self-Supervised Visual On-Policy Distillation](https://mlsanity.com/paper/2608.14144/) (2026)
- [Contrastive On-Policy Distillation](https://mlsanity.com/paper/2607.19046/) (2026)
- [Med-OPD: Improving Medical Vision-Language Models via Evidence-Aware On-Policy Distillation](https://mlsanity.com/paper/2607.16303/) (2026)
- [Perception Before Supervision: Self-Contained Visual Distillation from Counterfactual Blind Spots](https://mlsanity.com/paper/2608.09931/) (2026)
- [Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillation](https://mlsanity.com/paper/2609.23697/) (2026)
- [MMVistaReason: Toward Open-Data and Post-Training Recipes for Multimodal Reasoning](https://mlsanity.com/paper/2610.01352/) (2026)
- [Reward-Tilted On-Policy Distillation for Acoustic Grounding in Audio-Language Models](https://mlsanity.com/paper/2609.28778/) (2026)
- [Progressive-View On-Policy Distillation for Regional-to-Global Transfer in Multimodal LLMs](https://mlsanity.com/paper/2609.32333/) (2026)
- [SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Models](https://mlsanity.com/paper/2609.34044/) (2026)
- [Evidence-Aligned Multimodal On-Policy Self-Distillation for Fine-Grained Visual Understanding](https://mlsanity.com/paper/2609.34672/) (2026)
- [On-Policy Visual Evidence Distillation](https://mlsanity.com/paper/2609.36838/) (2026)
- [Spatial-OPSD: Self-Improving Spatial Reasoning via Label-Free Self-Distillation](https://mlsanity.com/paper/2609.37055/) (2026)
- [VISTA: Internalizing Collective Visual Experience via On-Policy Distillation for Active Multimodal Agents](https://mlsanity.com/paper/2609.38086/) (2026)
- [Distill the Visual Evidence, Not Just the Answer: Cross-World On-Policy Distillation for Vision-Language Models](https://mlsanity.com/paper/2609.38777/) (2026)
- [Is Better Teacher Supervision Enough? Unlocking Student-side Learning in Multimodal On-Policy Distillation](https://mlsanity.com/paper/2609.39120/) (2026)
- [LEGO-OPD: Factorized Teacher Composition for Multimodal On-Policy Distillation](https://mlsanity.com/paper/2610.00333/) (2026)
- [Beyond Entropy: Self-Diagnostic Multi-Role Token Optimization for Video Reasoning](https://mlsanity.com/paper/2610.03400/) (2026)
- [CV-OPD: Contrastive Vision On-policy Distillation](https://mlsanity.com/paper/2610.13264/) (2026)

## Abstract (arXiv)

On-policy knowledge distillation has proven effective for language models, yet its application to vision-language models (VLMs) remains underexplored. We observe that standard on-policy distillation can improve a student's output quality while failing to strengthen its reliance on visual input: on vision-critical tokens, the student's predictions remain largely unchanged whether or not fine-grained visual detail is present, even though the teacher's predictions depend heavily on it.To make this difference observable, we introduce visual advantage (VA), the token-level log-probability difference when the teacher scores a student-generated rollout with versus without access to fine-grained visual detail. VA is concentrated in a small minority of tokens, and these high-VA tokens are the ones that actually carry the visual supervision signal. This motivates a distillation objective that treats them differently from language scaffolding, so their contribution is not diluted by the abundant surrounding language tokens.We propose Visual-Advantage On-Policy Distillation (VA-OPD), which uses VA at two granularities: rollout-level reweighting by trajectory-averaged VA, and token-level KL averaged within high-VA and low-VA groups separately. We train on two math datasets (Geometry3K and ViRL39K) and evaluate on eight benchmarks covering both mathematical reasoning and visual understanding, across three teacher sizes (4B, 8B, and 32B) on the Qwen3-VL family. VA-OPD improves over standard on-policy distillation on every benchmark, with the gain growing monotonically along both the teacher-size and data-scale axes, suggesting that these factors compound consistently.