mlsanityData to 14 Oct 2026

Visual-Advantage On-Policy Distillation for Vision-Language Models

arXiv 2605.21924 ↗First posted 21 May 2026cs.CVcited by 24

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

On-policy knowledge distillation has proven effective for language models, yet its application to vision-language models (VLMs) remains underexplored. We observe that standard on-policy distillation can improve a student's output quality while failing to strengthen its reliance on visual input: on vision-critical tokens, the student's predictions remain largely unchanged whether or not fine-grained visual detail is present, even though the teacher's predictions depend heavily on it.To make this difference observable, we introduce visual advantage (VA), the token-level log-probability difference when the teacher scores a student-generated rollout with versus without access to fine-grained visual detail. VA is concentrated in a small minority of tokens, and these high-VA tokens are the ones that actually carry the visual supervision signal. This motivates a distillation objective that treats them differently from language scaffolding, so their contribution is not diluted by the abundant surrounding language tokens.We propose Visual-Advantage On-Policy Distillation (VA-OPD), which uses VA at two granularities: rollout-level reweighting by trajectory-averaged VA, and token-level KL averaged within high-VA and low-VA groups separately. We train on two math datasets (Geometry3K and ViRL39K) and evaluate on eight benchmarks covering both mathematical reasoning and visual understanding, across three teacher sizes (4B, 8B, and 32B) on the Qwen3-VL family. VA-OPD improves over standard on-policy distillation on every benchmark, with the gain growing monotonically along both the teacher-size and data-scale axes, suggesting that these factors compound consistently.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 32 references that are in the map. First posted on arXiv 21 May 2026.

Builds on

32 of 32 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Qwen3-VL Technical Report26 Nov

Other references · 31

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  4. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  5. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  6. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  7. 2016A Diagram Is Worth A Dozen Imagescited by 804
  8. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  9. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  10. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  11. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 559
  12. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 476
19 more
  1. 2015Policy Distillationcited by 440
  2. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 383
  3. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 323
  4. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 286
  5. 2024Multi-Modal Hallucination Control by Visual Information Groundingcited by 208
  6. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decodingcited by 203
  7. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 184
  8. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 175
  9. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 163
  10. 2025Perception-Aware Policy Optimization for Multimodal Reasoningcited by 99
  11. 2025Self-Rewarding Vision-Language Model via Reasoning Decompositioncited by 97
  12. 2024Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Samplingcited by 90
  13. 2025NoisyRollout: Reinforcing Visual Reasoning with Data Augmentationcited by 86
  14. 2024V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimizationcited by 67
  15. 2024Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Modelscited by 65
  16. 2025Spotlight on Token Perception for Multimodal Reinforcement Learningcited by 39
  17. 2024Visual Description Grounding Reduces Hallucinations and Boosts Reasoning in LVLMscited by 33
  18. 2026Rethinking Token-Level Policy Optimization for Multimodal Chain-of-Thoughtcited by 9
  19. 2024Token Preference Optimization with Self-Calibrated Visual-Anchored Rewards for Hallucination Mitigationcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 24.

24 papers.

Sort:

By area:

  1. 2026DOPD: Dual On-policy Distillationcited by 12
  2. 2026Visual Contrastive Self-Distillationcited by 10
  3. 2026Weak-to-Strong On-Policy Distillationcited by 8
  4. 2026OPD-V: Visual On-Policy Self-Distillation with Modality Balancecited by 8
  5. 2026VAD: Attributing Visual Evidence for Target Reconstruction in Multimodal On-Policy Distillationcited by 7
  6. 2026Latent On-Policy Self-Distillationcited by 7
  7. 2026Self-Supervised Visual On-Policy Distillationcited by 6
  8. 2026Contrastive On-Policy Distillationcited by 3
  9. 2026Med-OPD: Improving Medical Vision-Language Models via Evidence-Aware On-Policy Distillationcited by 2
  10. 2026Perception Before Supervision: Self-Contained Visual Distillation from Counterfactual Blind Spotscited by 2
  11. 2026Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillationcited by 2
  12. 2026MMVistaReason: Toward Open-Data and Post-Training Recipes for Multimodal Reasoningcited by 1
12 more
  1. 2026Reward-Tilted On-Policy Distillation for Acoustic Grounding in Audio-Language Modelscited by 0
  2. 2026Progressive-View On-Policy Distillation for Regional-to-Global Transfer in Multimodal LLMscited by 0
  3. 2026SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Modelscited by 0
  4. 2026Evidence-Aligned Multimodal On-Policy Self-Distillation for Fine-Grained Visual Understandingcited by 0
  5. 2026On-Policy Visual Evidence Distillationcited by 0
  6. 2026Spatial-OPSD: Self-Improving Spatial Reasoning via Label-Free Self-Distillationcited by 0
  7. 2026VISTA: Internalizing Collective Visual Experience via On-Policy Distillation for Active Multimodal Agentscited by 0
  8. 2026Distill the Visual Evidence, Not Just the Answer: Cross-World On-Policy Distillation for Vision-Language Modelscited by 0
  9. 2026Is Better Teacher Supervision Enough? Unlocking Student-side Learning in Multimodal On-Policy Distillationcited by 0
  10. 2026LEGO-OPD: Factorized Teacher Composition for Multimodal On-Policy Distillationcited by 0
  11. 2026Beyond Entropy: Self-Diagnostic Multi-Role Token Optimization for Video Reasoningcited by 0
  12. 2026CV-OPD: Contrastive Vision On-policy Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Distill What the Student Can See: Fisher-Projected On-Policy Distillation for Vision-Language Models
  2. 2026VC-OPD: Visual Counterfactual On-Policy Distillation for Grounded Vision-Language Reasoning
  3. 2026Decomposed On-Policy Distillation for Vision-Language Reasoning: Steering Gradients for Visual Grounding
  4. 2026On-Policy Distillation for Vision-Language Model Adaptation, an Effective Paradigm on Low-Quality Multimodal Data
  5. 2026Do As I Do, Not As I Say: On-Policy Distillation Inherits the Teacher in Vision-Language Models
  6. 2026CA-OPD: Confidence-Aware On-Policy Distillation for Structured Visual Prediction
  7. 2026KVE-KD: Key Visual Evidence-Guided Knowledge Distillation for Vision-Language Models
  8. 2026TTRSD: Test-Time Reinforcement Learning with Self-Distillation for Vision-Language Models