# Label-Guided Knowledge Distillation for 3D-CNNs in Action Recognition

arXiv: https://arxiv.org/abs/2609.13024 · HTML: https://mlsanity.com/paper/2609.13024/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 30 of its 42 references that are in the map.
- First posted on arXiv 11 Sep 2026.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 30)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [A Comprehensive Overhaul of Feature Distillation](https://mlsanity.com/paper/1904.01866/) (2019)
- [A Survey on Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2402.13116/) (2024)
- [Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition](https://mlsanity.com/paper/1708.07632/) (2017)
- [R-C3D: Region Convolutional 3D Network for Temporal Activity Detection](https://mlsanity.com/paper/1703.07814/) (2017)
- [Regularizing Class-wise Predictions via Self-knowledge Distillation](https://mlsanity.com/paper/2003.13964/) (2020)
- [Knowledge Distillation with the Reused Teacher Classifier](https://mlsanity.com/paper/2203.14001/) (2022)
- [A Survey of Visual Transformers](https://mlsanity.com/paper/2111.06091/) (2021)
- [Curriculum Temperature for Knowledge Distillation](https://mlsanity.com/paper/2211.16231/) (2022)
- [Masked Generative Distillation](https://mlsanity.com/paper/2205.01529/) (2022)
- [VideoPrism: A Foundational Visual Encoder for Video Understanding](https://mlsanity.com/paper/2402.13217/) (2024)
- [Logit Standardization in Knowledge Distillation](https://mlsanity.com/paper/2403.01427/) (2024)
- [Feature-map-level Online Adversarial Knowledge Distillation](https://mlsanity.com/paper/2002.01775/) (2020)
- [Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillation](https://mlsanity.com/paper/2103.08273/) (2021)
- [A Survey on Transformer Compression](https://mlsanity.com/paper/2402.05964/) (2024)
- [CLIP-guided Prototype Modulating for Few-shot Action Recognition](https://mlsanity.com/paper/2303.02982/) (2023)
- [Learning an Augmented RGB Representation with Cross-Modal Knowledge Distillation for Action Detection](https://mlsanity.com/paper/2108.03619/) (2021)
- [Action Sensitivity Learning for Temporal Action Localization](https://mlsanity.com/paper/2305.15701/) (2023)
- [Highlight Every Step: Knowledge Distillation via Collaborative Teaching](https://mlsanity.com/paper/1907.09643/) (2019)
- [Decomposed Cross-modal Distillation for RGB-based Temporal Action Detection](https://mlsanity.com/paper/2303.17285/) (2023)
- [ProKD: An Unsupervised Prototypical Knowledge Distillation Network for Zero-Resource Cross-Lingual Named Entity Recognition](https://mlsanity.com/paper/2301.08855/) (2023)

## Abstract (arXiv)

As a key model compression technique, knowledge distillation aims to transfer knowledge from a high-capacity teacher model to a lightweight student model for enhancing the latter's performance. In this work, we reviewed the feature knowledge distillation for 3D-CNNs and observed that most feature distillation methods in video analysis are simple adaptations of those used in image analysis, often neglecting the differences of video features in the temporal dimension. To address this issue, we proposed Label-Guided Knowledge Distillation (LGKD) to guide the distillation of student model features using ground truth labels. Our method entails two components: sample-wise distillation and class-wise distillation, enabling the student model to learn feature representation of the teacher model at two levels. Sample-wise distillation utilizes label information and the teacher's probability distribution to guide the learning of features that significantly impact temporal accuracy while mitigating noise. Meanwhile, class-wise feature distillation employs a prototype network to further capture the relational knowledge among samples within the same category, enhancing the student's ability to learn higher-dimensional semantic information and improving model generalization. To demonstrate the effectiveness and superiority of our method, we conducted comprehensive experiments on two benchmark action recognition datasets, UCF101 and HMDB51, achieving competitive results.