mlsanityData to 14 Oct 2026

Redirecting Teacher Supervision: Temporal-Response-Guided Latent Distillation for Video-Language Models

arXiv 2610.13045 ↗First posted 9 Oct 2026cs.CVcited by 0

On the map: Knowledge distillation → On-policy and self-distillation for LLMs also fits: Multimodal distillation

Abstract (arXiv)

On-policy distillation (OPD) transfers capabilities through teacher predictions on student-generated sequences, as studied by GKD and MiniLLM for language models and Video-OPD for video tasks. We ask how internal supervision can further support temporal learning. Direct hidden-state alignment does not explicitly use the student's temporal responses to choose a supervision direction. We propose Temporal-Response Guidance (TRG), which measures representation responses to local temporal perturbations, adds the teacher correction's projection onto their span back to the original correction, and restores its original norm. The projection is not normalized separately, and correction components outside the response space are retained. With an orthogonal tangent projector and a detached target, TRG changes the supervision direction while preserving the cosine-loss value and aligned-representation gradient norm at target construction. On TempCompass, Video-MME, MVBench, and Video-MMMU, TRG improves the 4B student's macro accuracy by 2.86 percentage points over the reported output-only OPD baseline.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 26 of its 29 references that are in the map. First posted on arXiv 9 Oct 2026.

Builds on

26 of 29 references.

References · 26

  1. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  2. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  3. 2021ViViT: A Video Vision Transformercited by 1,620
  4. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  5. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  6. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  7. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  8. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  9. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  10. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  11. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 902
  12. 2019Relational Knowledge Distillationcited by 889
14 more
  1. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 856
  2. 2016Sequence-Level Knowledge Distillationcited by 801
  3. 2019Contrastive Representation Distillationcited by 748
  4. 2019Similarity-Preserving Knowledge Distillationcited by 516
  5. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  6. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 484
  7. 2022Decoupled Knowledge Distillationcited by 388
  8. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 363
  9. 2019A Comprehensive Overhaul of Feature Distillationcited by 348
  10. 2019Variational Information Distillation for Knowledge Transfercited by 324
  11. 2021Distilling Knowledge via Knowledge Reviewcited by 288
  12. 2017Sobolev Training for Neural Networkscited by 143
  13. 2018Knowledge Transfer with Jacobian Matchingcited by 76
  14. 2026Video-OPD: Efficient Post-Training of Multimodal Large Language Models for Temporal Video Grounding via On-Policy Distillationcited by 36

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Less Data, Better Timing: Student-Curriculum Coupling for VLM On-Policy Distillation in Temporal Video Grounding
  2. 2026Context-Matched Distillation: Teacher Causality for Autoregressive Video Distillation
  3. 2026From Static to Dynamic: On-Policy Distillation from Image to Video Diffusion Models
  4. 2026When Teachers Mislead: Spurious-Signal-Aware On-Policy Distillation
  5. 2026Counterfactual Attention Policy Distillation for Temporal Video Grounding
  6. 2026Video-OPSD: Exploiting Privileged Visual Evidence for On-Policy Self-Distillation in Video Large Language Models
  7. 2026Video-MOPD: Multi-Teacher On-Policy Distillation for Video Understanding
  8. 2026Do As I Do, Not As I Say: On-Policy Distillation Inherits the Teacher in Vision-Language Models