mlsanityData to 14 Oct 2026

Multimodal distillation

308 papers2 this weekLow confidence in this grouping: the papers may belong to more than one topic, or the group may changeLow confidence in this grouping: the papers may belong to more than one topic, or the group may change · About

Keywords: distillation, visual, multimodal, knowledge, teacher

Key facts. Multimodal distillation: 308 papers on arXiv since 2021 (data through 14 Oct 2026). 25 of its 308 mapped papers (8%) cite “Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation” (2026, arXiv 2605.18740). 36 new papers 25 Sep–14 Oct.

What’s new

25 Sep – 14 Oct 2026

36 new papers, 25 Sep – 14 Oct; 2 this week.

  1. This week · 2
  2. LongForcing: Scaling Temporal Manifold Matching for Long-Horizon World Rollout
  3. DVTR-KD: Distilling Vision-Language Models via Intermediate Visual Token Reconstruction
  4. Earlier · 34
  5. VC-OPD: Visual Counterfactual On-Policy Distillation for Grounded Vision-Language Reasoning also fits: On-policy and self-distillation for LLMs
  6. CV-OPD: Contrastive Vision On-policy Distillation also fits: On-policy and self-distillation for LLMs
  7. Rubric-CEPR: Self-Evolving Image Editing via Reward-Verified Self-Distillation
  8. TAM: Task-Aware Memory Distillation for Efficient Spatiotemporal Prediction also fits: Detection and dense prediction distillation
  9. CARE: Constrained Attention Refinement for Fine-Grained Visual Classification via Teacher-Student Distillation also fits: Detection and dense prediction distillation
  10. Enhancing Long-Video VLM Embeddings with Query-Aware Streaming Latent Reasoning
  11. KVE-KD: Key Visual Evidence-Guided Knowledge Distillation for Vision-Language Models
  12. GRAFT: Growing Agglomerative Foundation Models via Continual Teacher Distillation also fits: General knowledge distillation methods
All 36
  1. Where-OPD: Spatially Guided On-Policy Self-Distillation of MLLMs with Synthetic Scenes
  2. Video Evidence Indexing: Learning Where to Look from Video Previews for Token-Budgeted Long-Video Question Answering
  3. Less Data, Better Timing: Student-Curriculum Coupling for VLM On-Policy Distillation in Temporal Video Grounding also fits: On-policy and self-distillation for LLMs
  4. Learning to Reason with Compressed Context: Ground-Truth-Free Adaptation of OmniLLMs via Self-Distillation also fits: Speech and audio distillation
  5. MCD: Causal Distillation of Multimodal In-Context Learning in Large Vision-Language Models
  6. OmniReasoning: Pushing the Limits of Audio-Visual Joint Reasoning
  7. Distill the Visual Evidence, Not Just the Answer: Cross-World On-Policy Distillation for Vision-Language Models
  8. UniEvo-VL: An On-policy Self-Distillation Training Recipe for Multimodal Model Self-improvement
  9. LEGO-OPD: Factorized Teacher Composition for Multimodal On-Policy Distillation
  10. LIFT: Layout-In-Future Video Generation under Large Viewpoint Change via On-Policy Self-Distillation
  11. VISTA: Internalizing Collective Visual Experience via On-Policy Distillation for Active Multimodal Agents also fits: On-policy and self-distillation for LLMs
  12. RS-OPSD: Reliable Privileged On-Policy-Self-Distillation for Ultra-High-Resolution Remote Sensing VQA also fits: Detection and dense prediction distillation
  13. Codebook-Guided Cross-Modal Knowledge Distillation for Structurally Heterogeneous Features
  14. Spatial-OPSD: Self-Improving Spatial Reasoning via Label-Free Self-Distillation
  15. EGSD: Event-Grounded Self-Distillation for Streaming Video Understanding also fits: On-policy and self-distillation for LLMs
  16. What Makes High-Magnification Knowledge Transferable? A Study of Cross-Resolution Distillation in Whole-Slide Imaging also fits: Detection and dense prediction distillation
  17. Compress to Remember: Learning Compact Memory via On-Policy Distillation for Long Video Generation also fits: Distillation for continual learning
  18. Evidence-Aligned Multimodal On-Policy Self-Distillation for Fine-Grained Visual Understanding
  19. Counterfactual Attention Policy Distillation for Temporal Video Grounding
  20. Distilling Visual Reasoning into Text Space
  21. K-OPSD: Verifiable On-Policy Self-Distillation for Post-Training Vision-Language Models on AEC Drawings
  22. SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Models
  23. TTRSD: Test-Time Reinforcement Learning with Self-Distillation for Vision-Language Models also fits: On-policy and self-distillation for LLMs
  24. Query, Align, and Distill: Navigation-Aware Cross-Modal Interaction for Efficient Vision-and-Language Navigation
  25. Fewer Tokens, More Self-Teaching: On-Policy Self-Distillation for Extreme Visual Token Reduction also fits: On-policy and self-distillation for LLMs
  26. Progressive-View On-Policy Distillation for Regional-to-Global Transfer in Multimodal LLMs

Foundations

Most cited by papers here

  1. Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation2026cited by 25 here
  2. CLIP-KD: An Empirical Study of CLIP Model Distillation2023cited by 12 here
  3. LLaVA-KD: A Framework of Distilling Multimodal Large Language Models2024cited by 11 here
  4. Compressing Visual-linguistic Model via Knowledge Distillation2021cited by 8 here
  5. The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillation2022cited by 8 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 48 here
  2. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 10 here
  3. 2024Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Samplingcited by 6 here
  4. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 25 here
  5. 2026Self-Distillation Enables Continual Learningcited by 10 here
  6. 2026Reinforcement Learning via Self-Distillationcited by 13 here
  7. 2026Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillationcited by 25 here
17 more it builds on, not in this chain
  1. 2021Compressing Visual-linguistic Model via Knowledge Distillationcited by 8 here
  2. 2021Distilling Knowledge via Knowledge Reviewcited by 6 here
  3. 2021Towards Understanding Knowledge Distillationcited by 6 here
  4. 2021Knowledge distillation: A good teacher is patient and consistentcited by 12 here
  5. 2021Online Knowledge Distillation for Efficient Pose Estimationcited by 6 here
  6. 2021Learning an Augmented RGB Representation with Cross-Modal Knowledge Distillation for Action Detectioncited by 6 here
  7. 2022Decoupled Knowledge Distillationcited by 17 here
  8. 2022Knowledge Distillation from A Stronger Teachercited by 9 here
  9. 2022The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillationcited by 8 here
  10. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 6 here
  11. 2023CLIP-KD: An Empirical Study of CLIP Model Distillationcited by 12 here
  12. 2024LLAVADI: What Matters For Multimodal Large Language Models Distillationcited by 6 here
  13. 2024LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillationcited by 12 here
  14. 2024LLaVA-KD: A Framework of Distilling Multimodal Large Language Modelscited by 11 here
  15. 2025MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoderscited by 7 here
  16. 2026Video-OPD: Efficient Post-Training of Multimodal Large Language Models for Temporal Video Grounding via On-Policy Distillationcited by 7 here
  17. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 10 here

The 24 most cited of 342.

Browse

30 of 308, ranked 6–35

  1. 6Video-OPD: Efficient Post-Training of Multimodal Large Language Models for Temporal Video Grounding via On-Policy Distillation20267
  2. 7MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders20257
  3. 8Visual-Advantage On-Policy Distillation for Vision-Language Models20266
  4. 9VOLD: Reasoning Transfer from LLMs to Vision-Language Models via On-Policy Distillation20256
  5. 10MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining20226
  6. 11Learning an Augmented RGB Representation with Cross-Modal Knowledge Distillation for Action Detection20216
  7. 12Decomposed On-Policy Distillation for Vision-Language Reasoning: Steering Gradients for Visual Grounding20265
  8. 13ViCuR: Visual Cues as Recoverable Privilege for Multimodal On-Policy Distillation20265
  9. 14Visual-OPSD: Cross-Modal On-Policy Self-Distillation for Efficient Unified Multimodal Reasoning20265
  10. 15V-Zero: Answer-Label-Free On-Policy Distillation with Contrastive Evidence Gating for Fine-Grained Visual Reasoning20265
All 30
  1. 16Decoupled Multimodal Distilling for Emotion Recognition20235
  2. 17Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation20225
  3. 18XKD: Cross-modal Knowledge Distillation with Domain Alignment for Video Representation Learning20225
  4. 19CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMs20254
  5. 20Active Data Curation Effectively Distills Large-Scale Multimodal Models20244
  6. 21Multimodal Distillation for Egocentric Action Recognition20234
  7. 22Learnable Cross-modal Knowledge Distillation for Multi-modal Learning with Missing Modality20234
  8. 23How many Observations are Enough? Knowledge Distillation for Trajectory Forecasting20224
  9. 24Multimodal Knowledge Expansion20214
  10. 25How to build a consistency model: Learning flow maps via self-distillation20253
  11. 26Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities20243
  12. 27Hydra-MDP: End-to-end Multimodal Planning with Multi-target Hydra-Distillation20243
  13. 28Improving Zero-shot Generalization of Learned Prompts via Unsupervised Knowledge Distillation20243
  14. 29CLIP-CID: Efficient CLIP Distillation via Cluster-Instance Discrimination20243
  15. 30DLIP: Distilling Language-Image Pre-training20233
  16. 31SILC: Improving Vision Language Pretraining with Self-Distillation20233
  17. 32A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations20233
  18. 33EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning20223
  19. 34VISD: Enhancing Video Reasoning via Structured Self-Distillation20262
  20. 35Visual Contrastive Self-Distillation20262