mlsanityData to 14 Oct 2026

OPD-Aha: From Linguistic Momentum to Visual Reflection in Multimodal On-Policy Distillation

arXiv 2609.16459 ↗First posted 15 Sep 2026cs.LG cs.AI cs.CVcited by 0

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Privileged on-policy distillation improves multimodal reasoning by allowing a teacher to evaluate student trajectories using rich, training-only visual evidence. Both models score these trajectories while conditioning on the same student-generated prefix. When a student misinterprets an image early in a response, this accumulating erroneous rationale eventually pulls the teacher away from its visual evidence. The teacher and student converge on the same hallucination, causing standard cross-model supervision to collapse precisely where correction is most needed. We find that the teacher's visual corrective preference is not lost under this misleading agreement. Comparing the predictions of the identical teacher given the real image and a visual null reveals that the privileged evidence still pushes the model toward the correct interpretation. We introduce OPD-Aha, which reconstructs the distillation target directly from this isolated visual preference rather than relying on the fragile teacher-student discrepancy. This reconstructed target aggressively suppresses continuations that contradict the image. Trained with this objective, students learn to naturally interrupt their own flawed reasoning with reflection tokens such as wait and actually. After reflection, subsequent generation relies less on the accumulated erroneous text and more on the visual evidence. Correcting these trajectories mid-generation fundamentally alters the reasoning process, yielding broad and consistent improvements across diverse fine-grained perception and complex multimodal reasoning benchmarks. Our code and models are available at https://github.com/Echochef/OPD-Aha.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 53 of its 58 references that are in the map. First posted on arXiv 15 Sep 2026.

Builds on

53 of 58 references.

Closest ancestors · 12

Same direction, newest first

  1. 2026Visual-OPSD: Cross-Modal On-Policy Self-Distillation for Efficient Unified Multimodal Reasoning17 Jun
  2. 2026Decomposed On-Policy Distillation for Vision-Language Reasoning: Steering Gradients for Visual Grounding30 May
  3. 2026Learn to Think: Improving Multimodal Reasoning through Vision-Aware Self-Improvement Training12 May
  4. 2026Video-OPD: Efficient Post-Training of Multimodal Large Language Models for Temporal Video Grounding via On-Policy Distillation3 Feb
  5. 2025VOLD: Reasoning Transfer from LLMs to Vision-Language Models via On-Policy Distillation27 Oct
  6. 2025SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning2 Jun
  7. 2025R1-Zero's "Aha Moment" in Visual Reasoning on a 2B Non-SFT Model7 Mar
  8. 2024Vision-Language Models Can Self-Improve Reasoning via Reflection30 Oct
  9. 2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models29 Oct
  10. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?1 Jul
  11. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset22 Feb
  12. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct

Other references · 41

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  7. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  8. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  9. 2023Teaching Large Language Models to Self-Debugcited by 988
  10. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  11. 2016Sequence-Level Knowledge Distillationcited by 801
  12. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
29 more
  1. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  2. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  3. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
  4. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 455
  5. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 307
  6. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 299
  7. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 292
  8. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 240
  9. 2015Unifying distillation and privileged informationcited by 233
  10. 2024Multi-Modal Hallucination Control by Visual Information Groundingcited by 208
  11. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 175
  12. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 163
  13. 2024Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Modelscited by 161
  14. 2026Privileged Information Distillation for Language Modelscited by 90
  15. 2026Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillationcited by 77
  16. 2026Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perceptioncited by 50
  17. 2024Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergencecited by 19
  18. 2026Pass the Baton: Trajectory-Relayed On-Policy Distillationcited by 18
  19. 2026Trajectory-Refined Distillationcited by 16
  20. 2024Pensieve: Retrospect-then-Compare Mitigates Visual Hallucinationcited by 14
  21. 2025Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMscited by 14
  22. 2026ViCuR: Visual Cues as Recoverable Privilege for Multimodal On-Policy Distillationcited by 12
  23. 2026Look Ahead Before You Distill: Future Trajectory Validation of Teacher Guidance for Agentic On-Policy Distillationcited by 11
  24. 2025The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs?cited by 9
  25. 2025Cross-Image Contrastive Decoding: Precise, Lossless Suppression of Language Priors in Large Vision-Language Modelscited by 8
  26. 2026Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understandingcited by 7
  27. 2026Residual Decoding: Mitigating Hallucinations in Large Vision-Language Models via History-Aware Residual Guidancecited by 5
  28. 2026Distill What the Student Can See: Fisher-Projected On-Policy Distillation for Vision-Language Modelscited by 4
  29. 2025LISA: A Layer-wise Integration and Suppression Approach for Hallucination Mitigation in Multimodal Large Language Modelscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026VAD: Attributing Visual Evidence for Target Reconstruction in Multimodal On-Policy Distillation
  2. 2026On-Policy Visual Evidence Distillation
  3. 2026Is Better Teacher Supervision Enough? Unlocking Student-side Learning in Multimodal On-Policy Distillation
  4. 2026LEGO-OPD: Factorized Teacher Composition for Multimodal On-Policy Distillation
  5. 2026Distilling Visual Reasoning into Text Space
  6. 2026Seeing Before Reasoning: Decoupling Perception and Reasoning for Shortcut-Resilient Multimodal On-Policy Self-Distillation
  7. 2026Distill the Visual Evidence, Not Just the Answer: Cross-World On-Policy Distillation for Vision-Language Models
  8. 2026Evidence-Aligned Multimodal On-Policy Self-Distillation for Fine-Grained Visual Understanding