# Generative Model-based Feature Knowledge Distillation for Action Recognition

arXiv: https://arxiv.org/abs/2312.08644 · HTML: https://mlsanity.com/paper/2312.08644/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 20 of its 32 references that are in the map.
- First posted on arXiv 14 Dec 2023.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 20 of 20)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [Decoupled Knowledge Distillation](https://mlsanity.com/paper/2203.08679/) (2022)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition](https://mlsanity.com/paper/1708.07632/) (2017)
- [Human Action Recognition from Various Data Modalities: A Review](https://mlsanity.com/paper/2012.11866/) (2020)
- [R-C3D: Region Convolutional 3D Network for Temporal Activity Detection](https://mlsanity.com/paper/1703.07814/) (2017)
- [Learning Salient Boundary Feature for Anchor-free Temporal Action Localization](https://mlsanity.com/paper/2103.13137/) (2021)
- [Focal and Global Knowledge Distillation for Detectors](https://mlsanity.com/paper/2111.11837/) (2021)
- [Knowledge Distillation via the Target-aware Transformer](https://mlsanity.com/paper/2205.10793/) (2022)
- [D3D: Distilled 3D Networks for Video Action Recognition](https://mlsanity.com/paper/1812.08249/) (2018)
- [Weakly-Supervised Action Localization by Generative Attention Modeling](https://mlsanity.com/paper/2003.12424/) (2020)
- [Cross-modal knowledge distillation for action recognition](https://mlsanity.com/paper/1910.04641/) (2019)
- [Semantics-aware Adaptive Knowledge Distillation for Sensor-to-Vision Action Recognition](https://mlsanity.com/paper/2009.00210/) (2020)
- [Learning an Augmented RGB Representation with Cross-Modal Knowledge Distillation for Action Detection](https://mlsanity.com/paper/2108.03619/) (2021)
- [Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamation](https://mlsanity.com/paper/1908.07121/) (2019)
- [System-status-aware Adaptive Network for Online Streaming Video Understanding](https://mlsanity.com/paper/2303.15742/) (2023)

## Built on by (most cited first, 7 of 7)

- [Event Stream based Human Action Recognition: A High-Definition Benchmark Dataset and Algorithms](https://mlsanity.com/paper/2408.09764/) (2024)
- [Sample-level Adaptive Knowledge Distillation for Action Recognition](https://mlsanity.com/paper/2504.00606/) (2025)
- [DVFL-Net: A Lightweight Distilled Video Focal Modulation Network for Spatio-Temporal Action Recognition](https://mlsanity.com/paper/2507.12426/) (2025)
- [Generative Model-Based Feature Attention Module for Video Action Analysis](https://mlsanity.com/paper/2508.13565/) (2025)
- [MobiAct: Efficient MAV Action Recognition Using MobileNetV4 with Contrastive Learning and Knowledge Distillation](https://mlsanity.com/paper/2510.19273/) (2025)
- [Scene-Aware Memory Discrimination: Deciding Which Personal Knowledge Stays](https://mlsanity.com/paper/2602.11607/) (2026)
- [Channel-wise Dynamic Knowledge Distillation via Adaptive Sample Generation for Action Recognition](https://mlsanity.com/paper/2608.03100/) (2026)

## Abstract (arXiv)

Knowledge distillation (KD), a technique widely employed in computer vision, has emerged as a de facto standard for improving the performance of small neural networks. However, prevailing KD-based approaches in video tasks primarily focus on designing loss functions and fusing cross-modal information. This overlooks the spatial-temporal feature semantics, resulting in limited advancements in model compression. Addressing this gap, our paper introduces an innovative knowledge distillation framework, with the generative model for training a lightweight student model. In particular, the framework is organized into two steps: the initial phase is Feature Representation, wherein a generative model-based attention module is trained to represent feature semantics; Subsequently, the Generative-based Feature Distillation phase encompasses both Generative Distillation and Attention Distillation, with the objective of transferring attention-based feature semantics with the generative model. The efficacy of our approach is demonstrated through comprehensive experiments on diverse popular datasets, proving considerable enhancements in video action recognition task. Moreover, the effectiveness of our proposed framework is validated in the context of more intricate video action detection task. Our code is available at https://github.com/aaai-24/Generative-based-KD.