Perception Before Supervision: Self-Contained Visual Distillation from Counterfactual Blind Spots
On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning
also in 1 more
Abstract (arXiv)
Self-improvement for multimodal large language models (MLLMs) is typically driven by reward-based methods that provide only coarse scalar feedback. Distillation offers a richer alternative through dense token-level supervision, but in the visual domain it usually depends on privileged context constructed using external annotations and tools, or stronger models. We introduce CVPD (Contrastive Counterfactual Visual Process Distillation), which, to the best of our knowledge, is the first fully self-contained framework for dense, on-policy, token-level visual self-distillation for MLLMs. CVPD identifies visual blind spots where zooming into a region changes and sharpens the model's answer distribution, while removing the same region leaves the full-image behavior largely unchanged. Such regions reveal perceptual information that the model can encode but fails to consistently utilize under full-image conditioning. We propose a three-gate Counterfactual Criterion that identifies these regions directly from the model's own responses and converts them into dense contrastive supervision for self-distillation. On Qwen3-VL-8B-Instruct, CVPD outperforms six self-evolving baselines across twelve benchmarks, including methods that rely on external GPT-4o supervision, without a single regression. It achieves gains of +3.60 on OCRBench, +3.38 on MMStar Fine-Grained Perception, and +3.08 on MMStar Logical Reasoning, while maintaining or improving performance on broader multimodal benchmarks.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 42 references that are in the map. First posted on arXiv 10 Aug 2026.
Builds on
34 of 42 references.
Closest ancestors · 6
Same direction, newest first
- 2026Visual-Advantage On-Policy Distillation for Vision-Language Models21 May
- 2026Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation18 May
- 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
- 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
- 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
- 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
Other references · 28
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
- 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
- 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
- 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
- 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
- 2016A Diagram Is Worth A Dozen Imagescited by 804
- 2024Self-Rewarding Language Modelscited by 681
- 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 526
- 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
- 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 476
16 more
- 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 455
- 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 301
- 2025R1-ShareVL: Incentivizing Reasoning Capability of Multimodal Large Language Models via Share-GRPOcited by 52
- 2025VisPlay: Self-Evolving Vision-Language Models from Imagescited by 47
- 2025Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Playcited by 37
- 2025EvoLMM: Self-Evolving Large Multimodal Models with Continuous Rewardscited by 27
- 2026MM-Zero: Self-Evolving Multi-Model Vision Language Models From Zero Datacited by 21
- 2026ROSD: Reflective On-Policy Self-Distillation for Language Model Reasoning across Domainscited by 17
- 2026OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoningcited by 16
- 2026Active Zero: Self-Evolving Vision-Language Models through Active Environment Explorationcited by 8
- 2026CEPO: RLVR Self-Distillation using Contrastive Evidence Policy Optimizationcited by 8
- 2026iReasoner: Trajectory-Aware Intrinsic Reasoning Supervision for Self-Evolving Large Multimodal Modelscited by 7
- 2026Ask, Solve, Generate: Self-Evolving Unified Multimodal Understanding and Generation via Self-Consistency Rewardscited by 7
- 2026Paying More Attention to Visual Tokens in Self-Evolving Large Multimodal Modelscited by 6
- 2026When Models Judge Themselves: Unsupervised Self-Evolution for Multimodal Reasoningcited by 5
- 2026RISE: Reliable Improvement in Self-Evolving Vision-Language Modelscited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Seeing Before Reasoning: Decoupling Perception and Reasoning for Shortcut-Resilient Multimodal On-Policy Self-Distillation
- 2026Fewer Tokens, More Self-Teaching: On-Policy Self-Distillation for Extreme Visual Token Reduction
- 2026OPD-V: Visual On-Policy Self-Distillation with Modality Balance
- 2026Visual Contrastive Self-Distillation
- 2026VC-OPD: Visual Counterfactual On-Policy Distillation for Grounded Vision-Language Reasoning
- 2026Boosting Visual Instruction Tuning with Self-Supervised Guidance
- 2026BAS-OPD: Budget-Aware Selective On-Policy Self-Distillation for Fine-Grained Multimodal Perception
- 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining