# D3D: Distilled 3D Networks for Video Action Recognition

arXiv: https://arxiv.org/abs/1812.08249 · HTML: https://mlsanity.com/paper/1812.08249/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Dec 2018.

## Built on by (most cited first, 30 of 60)

- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text](https://mlsanity.com/paper/2104.11178/) (2021)
- [ActionCLIP: A New Paradigm for Video Action Recognition](https://mlsanity.com/paper/2109.08472/) (2021)
- [TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?](https://mlsanity.com/paper/2106.11297/) (2021)
- [Broaden Your Views for Self-Supervised Video Learning](https://mlsanity.com/paper/2103.16559/) (2021)
- [Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers](https://mlsanity.com/paper/2103.16553/) (2021)
- [Vita-CLIP: Video and text adaptive CLIP via Multimodal Prompting](https://mlsanity.com/paper/2304.03307/) (2023)
- [ACTION-Net: Multipath Excitation for Action Recognition](https://mlsanity.com/paper/2103.07372/) (2021)
- [Towards Good Practices for Missing Modality Robust Action Recognition](https://mlsanity.com/paper/2211.13916/) (2022)
- [MM-ViT: Multi-Modal Video Transformer for Compressed Video Action Recognition](https://mlsanity.com/paper/2108.09322/) (2021)
- [MMANet: Margin-aware Distillation and Modality-aware Regularization for Incomplete Multimodal Learning](https://mlsanity.com/paper/2304.08028/) (2023)
- [Analysis of Video Quality Datasets via Design of Minimalistic Video Quality Models](https://mlsanity.com/paper/2307.13981/) (2023)
- [Disentangling Architecture and Training for Optical Flow](https://mlsanity.com/paper/2203.10712/) (2022)
- [Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications](https://mlsanity.com/paper/2204.11786/) (2022)
- [Multimodal Distillation for Egocentric Action Recognition](https://mlsanity.com/paper/2307.07483/) (2023)
- [Learning from Temporal Gradient for Semi-supervised Action Recognition](https://mlsanity.com/paper/2111.13241/) (2021)
- [Motion-Focused Contrastive Learning of Video Representations](https://mlsanity.com/paper/2201.04029/) (2022)
- [Bodily Behaviors in Social Interaction: Novel Annotations and State-of-the-Art Evaluation](https://mlsanity.com/paper/2207.12817/) (2022)
- [Egocentric Video Task Translation](https://mlsanity.com/paper/2212.06301/) (2022)
- [Motion-Augmented Self-Training for Video Recognition at Smaller Scale](https://mlsanity.com/paper/2105.01646/) (2021)
- [DnS: Distill-and-Select for Efficient and Accurate Video Indexing and Retrieval](https://mlsanity.com/paper/2106.13266/) (2021)
- [End-to-End Semi-Supervised Learning for Video Action Detection](https://mlsanity.com/paper/2203.04251/) (2022)
- [Vision-Language Models can Identify Distracted Driver Behavior from Naturalistic Videos](https://mlsanity.com/paper/2306.10159/) (2023)
- [Synchronization is All You Need: Exocentric-to-Egocentric Transfer for Temporal Action Segmentation with Unlabeled Synchronized Video Pairs](https://mlsanity.com/paper/2312.02638/) (2023)
- [Robust Multimodal Learning via Representation Decoupling](https://mlsanity.com/paper/2407.04458/) (2024)
- [Computation-efficient Deep Learning for Computer Vision: A Survey](https://mlsanity.com/paper/2308.13998/) (2023)
- [Modality Mixer for Multi-modal Action Recognition](https://mlsanity.com/paper/2208.11314/) (2022)
- [Rethinking Resolution in the Context of Efficient Video Recognition](https://mlsanity.com/paper/2209.12797/) (2022)
- [Compressed Vision for Efficient Video Understanding](https://mlsanity.com/paper/2210.02995/) (2022)
- [Generative Model-based Feature Knowledge Distillation for Action Recognition](https://mlsanity.com/paper/2312.08644/) (2023)

## Abstract (arXiv)

State-of-the-art methods for video action recognition commonly use an ensemble of two networks: the spatial stream, which takes RGB frames as input, and the temporal stream, which takes optical flow as input. In recent work, both of these streams consist of 3D Convolutional Neural Networks, which apply spatiotemporal filters to the video clip before performing classification. Conceptually, the temporal filters should allow the spatial stream to learn motion representations, making the temporal stream redundant. However, we still see significant benefits in action recognition performance by including an entirely separate temporal stream, indicating that the spatial stream is "missing" some of the signal captured by the temporal stream. In this work, we first investigate whether motion representations are indeed missing in the spatial stream of 3D CNNs. Second, we demonstrate that these motion representations can be improved by distillation, by tuning the spatial stream to predict the outputs of the temporal stream, effectively combining both models into a single stream. Finally, we show that our Distilled 3D Network (D3D) achieves performance on par with two-stream approaches, using only a single model and with no need to compute optical flow.