# VOLD: Reasoning Transfer from LLMs to Vision-Language Models via On-Policy Distillation

arXiv: https://arxiv.org/abs/2510.23497 · HTML: https://mlsanity.com/paper/2510.23497/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 21 of its 47 references that are in the map.
- First posted on arXiv 27 Oct 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)
- Knowledge distillation / [Multimodal distillation](https://mlsanity.com/distill/multimodal-distillation/)

## Builds on (most cited first, 21 of 21)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [Group Sequence Policy Optimization](https://mlsanity.com/paper/2507.18071/) (2025)
- [Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models](https://mlsanity.com/paper/2503.06749/) (2025)
- [VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model](https://mlsanity.com/paper/2504.07615/) (2025)
- [Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model](https://mlsanity.com/paper/2503.24290/) (2025)
- [R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalization](https://mlsanity.com/paper/2503.10615/) (2025)
- [We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?](https://mlsanity.com/paper/2407.01284/) (2024)
- [LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts](https://mlsanity.com/paper/2407.04973/) (2024)
- [SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models](https://mlsanity.com/paper/2504.11468/) (2025)
- [KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning](https://mlsanity.com/paper/2506.02208/) (2025)
- [DCPO: Dynamic Clipping Policy Optimization](https://mlsanity.com/paper/2509.02333/) (2025)
- [X-Reasoner: Towards Generalizable Reasoning Across Modalities and Domains](https://mlsanity.com/paper/2505.03981/) (2025)

## Built on by (most cited first, 30 of 31)

- [A Survey of On-Policy Distillation for Large Language Models](https://mlsanity.com/paper/2604.00626/) (2026)
- [Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe](https://mlsanity.com/paper/2605.03677/) (2026)
- [Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillation](https://mlsanity.com/paper/2606.02684/) (2026)
- [SOD: Step-wise On-policy Distillation for Small Language Model Agents](https://mlsanity.com/paper/2605.07725/) (2026)
- [Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation](https://mlsanity.com/paper/2605.13643/) (2026)
- [Guava: Distilling Frontier VLMs into a Compact Agent through a Robotic Manipulation Harness](https://mlsanity.com/paper/2606.18363/) (2026)
- [Visual Contrastive Self-Distillation](https://mlsanity.com/paper/2607.21556/) (2026)
- [Beyond SFT-to-RL: Pre-alignment via Black-Box On-Policy Distillation for Multimodal RL](https://mlsanity.com/paper/2604.28123/) (2026)
- [SenseNova-U1.5: Towards Native Unified Visual Intelligence](https://mlsanity.com/paper/2609.11929/) (2026)
- [A Formula-Driven Survey and Research Agenda for On-Policy Distillation](https://mlsanity.com/paper/2606.22793/) (2026)
- [Self-Supervised Visual On-Policy Distillation](https://mlsanity.com/paper/2608.14144/) (2026)
- [Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradients](https://mlsanity.com/paper/2606.18216/) (2026)
- [Learning Visual Spatial Planning from Symbolic State via Modality-Gap-Aware Self-Distillation](https://mlsanity.com/paper/2606.06076/) (2026)
- [Deep Thought Alignment: Trajectory-Level Latent Distillation for Video Reasoning](https://mlsanity.com/paper/2608.16316/) (2026)
- [KEPO: Knowledge-Enhanced Preference Optimization for Multimodal Reasoning with Applications to Medical VQA](https://mlsanity.com/paper/2602.00400/) (2026)
- [Stage-1 Controls the Entropy Regime, Not the Outcome](https://mlsanity.com/paper/2606.09059/) (2026)
- [MIRROR: Learning from the Other View for Multi-Modal Reasoning](https://mlsanity.com/paper/2607.21552/) (2026)
- [OracleZoom: On-Policy Self-Distillation Inspired Reference-Constrained Recursive Image Super Resolution](https://mlsanity.com/paper/2609.06490/) (2026)
- [MMVistaReason: Toward Open-Data and Post-Training Recipes for Multimodal Reasoning](https://mlsanity.com/paper/2610.01352/) (2026)
- [Unifying ICL, SFT, KL-Regularized RL Through a Bayesian Lens](https://mlsanity.com/paper/2609.05111/) (2026)
- [OPD-Aha: From Linguistic Momentum to Visual Reflection in Multimodal On-Policy Distillation](https://mlsanity.com/paper/2609.16459/) (2026)
- [From Self-Distillation to Self-Practice: Privileged Information for Multi-Turn Agents](https://mlsanity.com/paper/2609.29051/) (2026)
- [USA: Update-aware SAM for Cross-domain On-Policy Disitllation of Language Agents](https://mlsanity.com/paper/2609.34225/) (2026)
- [Look Before You Select: Rethinking Vocabulary Sparsification in On-Policy Distillation](https://mlsanity.com/paper/2609.34386/) (2026)
- [Counterfactual Attention Policy Distillation for Temporal Video Grounding](https://mlsanity.com/paper/2609.34581/) (2026)
- [EGSD: Event-Grounded Self-Distillation for Streaming Video Understanding](https://mlsanity.com/paper/2609.36803/) (2026)
- [Is Better Teacher Supervision Enough? Unlocking Student-side Learning in Multimodal On-Policy Distillation](https://mlsanity.com/paper/2609.39120/) (2026)
- [Same Reward, Different Skills: When Multimodal RL Learns to Look](https://mlsanity.com/paper/2610.01908/) (2026)
- [Text-Centric Post-Training for Omni-Modal Reasoning](https://mlsanity.com/paper/2610.02819/) (2026)
- [SCAD: Structured Credit Assignment and Distillation for Long-Horizon Agents](https://mlsanity.com/paper/2610.03372/) (2026)

## Abstract (arXiv)

Training vision-language models (VLMs) for complex reasoning remains a challenging task, i.a. due to the scarcity of high-quality image-text reasoning data. Conversely, text-based reasoning resources are abundant and scalable, but it is still an open question how to leveraging them for VLM reasoning. To address this problem, we propose VOLD, a framework to transfer reasoning capabilities from text-only teacher models to VLM student models. To this end, VOLD combines reinforcement learning via Group Relative Policy Optimization (GRPO) with on-policy distillation, which allows the student reasoning traces to be guided by the teacher model, resulting in a significant gain over using GRPO alone. We further show that a cold-start alignment is essential for an effective transfer during the online training phase in this scenario and that without sufficient distributional alignment between teacher and student, on-policy distillation fails to provide meaningful guidance. We evaluate VOLD across diverse benchmarks including MMMU-Pro, MathVision, MathVista, and LogicVista, showing that VOLD outperforms the baseline model significantly and improves over the state of the art by a margin. Our ablation shows the importance of a cold-start alignment via SFT for on-policy distillation with a text-only teacher.