VC-OPD: Visual Counterfactual On-Policy Distillation for Grounded Vision-Language Reasoning
On the map: LLM reasoning → Multimodal reasoning
also in 1 more
Abstract (arXiv)
GRPO-style reinforcement learning has advanced vision-language reasoning, but its sequence-level rewards provide limited credit assignment for individual reasoning tokens. On-policy distillation (OPD) addresses this issue by providing dense supervision on student-generated trajectories, yet standard OPD assigns similar importance to all teacher-preferred tokens, including template phrases and visually irrelevant continuations. We introduce Visual Counterfactual On-Policy Distillation (VC-OPD), a grounding-aware distillation method that prioritizes tokens whose correctness depends on image-specific evidence. After a teacher-supervised warm start that stabilizes response formats and reduces noisy rollouts, VC-OPD performs on-policy distillation with two teacher evaluations for each student trajectory: one under the original image and one under a counterfactual image with corrupted instance-specific evidence. The original-image teacher distribution determines the distillation target, while the original--counterfactual teacher gap estimates token-level visual dependence and softly reweights the loss. To avoid unstable response-level update scales, VC-OPD further applies mass-preserving normalization to the token weights. This design preserves the dense supervision of OPD while shifting learning capacity toward visually grounded reasoning tokens. Experiments on multimodal reasoning benchmarks show that VC-OPD improves visual reasoning performance and produces more effective token-level distillation.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 36 of its 37 references that are in the map. First posted on arXiv 10 Oct 2026.
Builds on
36 of 37 references.
Closest ancestors · 15
Same direction, newest first
- 2025Perception Before Reasoning: Two-Stage Reinforcement Learning for Visual Reasoning in Vision-Language Models16 Sep
- 2025Self-Rewarding Vision-Language Model via Reasoning Decomposition27 Aug
- 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency25 Aug
- 2025Thyme: Think Beyond Images15 Aug
- 2025Perception-Aware Policy Optimization for Multimodal Reasoning8 Jul
- 2025Grounded Reinforcement Learning for Visual Reasoning29 May
- 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning20 May
- 2025Visual Agentic Reinforcement Fine-Tuning20 May
- 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning10 Apr
- 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model10 Apr
- 2025Visual-RFT: Visual Reinforcement Fine-Tuning3 Mar
- 2024LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts6 Jul
3 more
Other references · 21
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2025Qwen3-VL Technical Reportcited by 2,793
- 2015Trust Region Policy Optimizationcited by 2,580
- 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
- 2016Sequence-Level Knowledge Distillationcited by 801
- 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
- 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
- 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
- 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 412
9 more
- 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 383
- 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 301
- 2026On-Policy Context Distillation for Language Modelscited by 186
- 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 184
- 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 175
- 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 83
- 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillationcited by 82
- 2026Fast and Effective On-policy Distillation from Reasoning Prefixescited by 53
- 2025Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedbackcited by 30
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Decomposed On-Policy Distillation for Vision-Language Reasoning: Steering Gradients for Visual Grounding
- 2026LEGO-OPD: Factorized Teacher Composition for Multimodal On-Policy Distillation
- 2026CT-OPD: Counterfactual Trace On-Policy Distillation for Diffusion Vision-Language Models
- 2026Distill What the Student Can See: Fisher-Projected On-Policy Distillation for Vision-Language Models
- 2026V-Zero: Answer-Label-Free On-Policy Distillation with Contrastive Evidence Gating for Fine-Grained Visual Reasoning
- 2026Seeing Before Reasoning: Decoupling Perception and Reasoning for Shortcut-Resilient Multimodal On-Policy Self-Distillation
- 2026OPD-Aha: From Linguistic Momentum to Visual Reflection in Multimodal On-Policy Distillation
- 2026Visual-Advantage On-Policy Distillation for Vision-Language Models