# VAD: Attributing Visual Evidence for Target Reconstruction in Multimodal On-Policy Distillation

arXiv: https://arxiv.org/abs/2607.28590 · HTML: https://mlsanity.com/paper/2607.28590/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal distillation, one of 12 directions in Knowledge distillation.
- Builds on 32 of its 46 references that are in the map.
- First posted on arXiv 30 Jul 2026.

## Placed in

- Knowledge distillation / [Multimodal distillation](https://mlsanity.com/distill/multimodal-distillation/)

## Builds on (most cited first, 30 of 32)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [Are We on the Right Way for Evaluating Large Vision-Language Models?](https://mlsanity.com/paper/2403.20330/) (2024)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding](https://mlsanity.com/paper/2311.16922/) (2023)
- [V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs](https://mlsanity.com/paper/2312.14135/) (2023)
- [Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models](https://mlsanity.com/paper/2403.18814/) (2024)
- [MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?](https://mlsanity.com/paper/2408.13257/) (2024)
- [LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images](https://mlsanity.com/paper/2403.11703/) (2024)
- [On-Policy Context Distillation for Language Models](https://mlsanity.com/paper/2602.12275/) (2026)
- [HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding](https://mlsanity.com/paper/2403.00425/) (2024)
- [DistiLLM: Towards Streamlined Distillation for Large Language Models](https://mlsanity.com/paper/2402.03898/) (2024)
- [DeepEyesV2: Toward Agentic Multimodal Model](https://mlsanity.com/paper/2511.05271/) (2025)
- [ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Exploration](https://mlsanity.com/paper/2411.16044/) (2024)
- [DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs](https://mlsanity.com/paper/2503.07067/) (2025)
- [ReFocus: Visual Editing as a Chain of Thought for Structured Image Understanding](https://mlsanity.com/paper/2501.05452/) (2025)
- [Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation](https://mlsanity.com/paper/2605.18740/) (2026)
- [Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perception](https://mlsanity.com/paper/2602.11858/) (2026)
- [SenseNova-MARS: Empowering Multimodal Agentic Reasoning and Search via Reinforcement Learning](https://mlsanity.com/paper/2512.24330/) (2025)
- [Visual-Advantage On-Policy Distillation for Vision-Language Models](https://mlsanity.com/paper/2605.21924/) (2026)
- [Visual-OPSD: Cross-Modal On-Policy Self-Distillation for Efficient Unified Multimodal Reasoning](https://mlsanity.com/paper/2606.18974/) (2026)
- [Delve into Visual Contrastive Decoding for Hallucination Mitigation of Large Vision-Language Models](https://mlsanity.com/paper/2412.06775/) (2024)
- [ViCuR: Visual Cues as Recoverable Privilege for Multimodal On-Policy Distillation](https://mlsanity.com/paper/2606.05718/) (2026)
- [V-Zero: Answer-Label-Free On-Policy Distillation with Contrastive Evidence Gating for Fine-Grained Visual Reasoning](https://mlsanity.com/paper/2606.25319/) (2026)
- [Decomposed On-Policy Distillation for Vision-Language Reasoning: Steering Gradients for Visual Grounding](https://mlsanity.com/paper/2606.00564/) (2026)
- [Thinking with Deltas: Incentivizing Reinforcement Learning via Differential Visual Reasoning Policy](https://mlsanity.com/paper/2601.06801/) (2026)
- [Beyond Absolute Imitation: Anchored Residual Guidance for Privileged On-Policy Distillation](https://mlsanity.com/paper/2606.10385/) (2026)

## Built on by (most cited first, 7 of 7)

- [Learning from Unreachable Rewards: Hint-Conditioned Reinforcement Learning for Generative Recommendation](https://mlsanity.com/paper/2608.11980/) (2026)
- [Evidence-Aligned Multimodal On-Policy Self-Distillation for Fine-Grained Visual Understanding](https://mlsanity.com/paper/2609.34672/) (2026)
- [On-Policy Visual Evidence Distillation](https://mlsanity.com/paper/2609.36838/) (2026)
- [Distill the Visual Evidence, Not Just the Answer: Cross-World On-Policy Distillation for Vision-Language Models](https://mlsanity.com/paper/2609.38777/) (2026)
- [Is Better Teacher Supervision Enough? Unlocking Student-side Learning in Multimodal On-Policy Distillation](https://mlsanity.com/paper/2609.39120/) (2026)
- [Slow-Fast Multi-Teacher On-Policy Distillation for Capability Preservation](https://mlsanity.com/paper/2610.02324/) (2026)
- [CV-OPD: Contrastive Vision On-policy Distillation](https://mlsanity.com/paper/2610.13264/) (2026)

## Abstract (arXiv)

Multimodal on-policy distillation (OPD) transfers fine-grained visual knowledge by supervising student-generated trajectories with a privileged-view teacher. Yet its next-token corrections are source-mixed, combining visual signals with linguistic priors and teacher-specific effects. The key challenge is to estimate which corrections are supported by visual evidence, not merely where or how strongly to distill. We introduce Visual Attribution Distillation (VAD), a counterfactual target-reconstruction algorithm that estimates the visually attributable part of a teacher correction. At each student-generated prefix, VAD evaluates the same fixed teacher with the relevant evidence present and removed. The corresponding change in centered log-probabilities defines ut, a signed proxy for the visual evidence direction that estimates how revealing the evidence supports or refutes candidate tokens. VAD projects the original correction onto this proxy to obtain an intervention-aligned component and a proxy-unexplained residual, then reconstructs a student-anchored target from the former. During training, this reconstructed target supplies the primary supervision signal, while the privileged teacher contributes a weak regularizer. Across six fine-grained visual benchmarks at 4B and 9B scales, VAD outperforms direct privileged-view distillation and visual-advantage weighting. Token- level and controlled-target analyses show that the proxy-aligned component is enriched in task-relevant visual corrections and yields stronger target shifts, especially when evidence refutes a mistaken answer. These results support counterfactual target reconstruction as an effective alternative to source-mixed supervision.