mlsanityData to 14 Oct 2026

VAD: Attributing Visual Evidence for Target Reconstruction in Multimodal On-Policy Distillation

arXiv 2607.28590 ↗First posted 30 Jul 2026cs.CV cs.CLcited by 7

On the map: Knowledge distillation → Multimodal distillation

Abstract (arXiv)

Multimodal on-policy distillation (OPD) transfers fine-grained visual knowledge by supervising student-generated trajectories with a privileged-view teacher. Yet its next-token corrections are source-mixed, combining visual signals with linguistic priors and teacher-specific effects. The key challenge is to estimate which corrections are supported by visual evidence, not merely where or how strongly to distill. We introduce Visual Attribution Distillation (VAD), a counterfactual target-reconstruction algorithm that estimates the visually attributable part of a teacher correction. At each student-generated prefix, VAD evaluates the same fixed teacher with the relevant evidence present and removed. The corresponding change in centered log-probabilities defines ut, a signed proxy for the visual evidence direction that estimates how revealing the evidence supports or refutes candidate tokens. VAD projects the original correction onto this proxy to obtain an intervention-aligned component and a proxy-unexplained residual, then reconstructs a student-anchored target from the former. During training, this reconstructed target supplies the primary supervision signal, while the privileged teacher contributes a weak regularizer. Across six fine-grained visual benchmarks at 4B and 9B scales, VAD outperforms direct privileged-view distillation and visual-advantage weighting. Token- level and controlled-target analyses show that the proxy-aligned component is enriched in task-relevant visual corrections and yields stronger target shifts, especially when evidence refutes a mistaken answer. These results support counterfactual target reconstruction as an effective alternative to source-mixed supervision.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Multimodal distillation, one of 12 directions in Knowledge distillation. Builds on 32 of its 46 references that are in the map. First posted on arXiv 30 Jul 2026.

Builds on

32 of 46 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026Med-OPD: Improving Medical Vision-Language Models via Evidence-Aware On-Policy Distillation14 Jul
  2. 2026V-Zero: Answer-Label-Free On-Policy Distillation with Contrastive Evidence Gating for Fine-Grained Visual Reasoning24 Jun
  3. 2026Visual-OPSD: Cross-Modal On-Policy Self-Distillation for Efficient Unified Multimodal Reasoning17 Jun
  4. 2026ViCuR: Visual Cues as Recoverable Privilege for Multimodal On-Policy Distillation4 Jun
  5. 2026Decomposed On-Policy Distillation for Vision-Language Reasoning: Steering Gradients for Visual Grounding30 May
  6. 2026Visual-Advantage On-Policy Distillation for Vision-Language Models21 May
  7. 2026Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation18 May

Other references · 25

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  5. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  6. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  7. 2016Sequence-Level Knowledge Distillationcited by 801
  8. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  9. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
  10. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  11. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 292
  12. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 195
13 more
  1. 2026On-Policy Context Distillation for Language Modelscited by 186
  2. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 174
  3. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 163
  4. 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 125
  5. 2024ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Explorationcited by 103
  6. 2025DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMscited by 84
  7. 2025ReFocus: Visual Editing as a Chain of Thought for Structured Image Understandingcited by 83
  8. 2026Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perceptioncited by 50
  9. 2025SenseNova-MARS: Empowering Multimodal Agentic Reasoning and Search via Reinforcement Learningcited by 33
  10. 2024Delve into Visual Contrastive Decoding for Hallucination Mitigation of Large Vision-Language Modelscited by 12
  11. 2026Thinking with Deltas: Incentivizing Reinforcement Learning via Differential Visual Reasoning Policycited by 6
  12. 2026Beyond Absolute Imitation: Anchored Residual Guidance for Privileged On-Policy Distillationcited by 3
  13. 2026Contrastive On-Policy Distillationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026Learning from Unreachable Rewards: Hint-Conditioned Reinforcement Learning for Generative Recommendationcited by 1
  2. 2026Evidence-Aligned Multimodal On-Policy Self-Distillation for Fine-Grained Visual Understandingcited by 0
  3. 2026On-Policy Visual Evidence Distillationcited by 0
  4. 2026Distill the Visual Evidence, Not Just the Answer: Cross-World On-Policy Distillation for Vision-Language Modelscited by 0
  5. 2026Is Better Teacher Supervision Enough? Unlocking Student-side Learning in Multimodal On-Policy Distillationcited by 0
  6. 2026Slow-Fast Multi-Teacher On-Policy Distillation for Capability Preservationcited by 0
  7. 2026CV-OPD: Contrastive Vision On-policy Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026OPD-Aha: From Linguistic Momentum to Visual Reflection in Multimodal On-Policy Distillation
  2. 2026LEGO-OPD: Factorized Teacher Composition for Multimodal On-Policy Distillation
  3. 2026Distill What the Student Can See: Fisher-Projected On-Policy Distillation for Vision-Language Models
  4. 2026VC-OPD: Visual Counterfactual On-Policy Distillation for Grounded Vision-Language Reasoning
  5. 2026MCD: Causal Distillation of Multimodal In-Context Learning in Large Vision-Language Models
  6. 2026Self-Supervised Visual On-Policy Distillation
  7. 2026Tracing the Evidence: Faithful Token Attribution Through Vision-Language Reasoning
  8. 2026CA-OPD: Confidence-Aware On-Policy Distillation for Structured Visual Prediction