# TEAM-Net: Multi-modal Learning for Video Action Recognition with Partial Decoding

arXiv: https://arxiv.org/abs/2110.08814 · HTML: https://mlsanity.com/paper/2110.08814/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 48 references that are in the map.
- First posted on arXiv 17 Oct 2021.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [What Makes Training Multi-Modal Classification Networks Hard?](https://mlsanity.com/paper/1905.12681/) (2019)
- [EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition](https://mlsanity.com/paper/1908.08498/) (2019)
- [Use What You Have: Video Retrieval Using Representations From Collaborative Experts](https://mlsanity.com/paper/1907.13487/) (2019)
- [TEA: Temporal Excitation and Aggregation for Action Recognition](https://mlsanity.com/paper/2004.01398/) (2020)
- [STM: SpatioTemporal and Motion Encoding for Action Recognition](https://mlsanity.com/paper/1908.02486/) (2019)
- [MMTM: Multimodal Transfer Module for CNN Fusion](https://mlsanity.com/paper/1911.08670/) (2019)
- [Compressed Video Action Recognition](https://mlsanity.com/paper/1712.00636/) (2017)
- [2D/3D Pose Estimation and Action Recognition using Multitask Deep Learning](https://mlsanity.com/paper/1802.09232/) (2018)
- [Learning a Text-Video Embedding from Incomplete and Heterogeneous Data](https://mlsanity.com/paper/1804.02516/) (2018)
- [Involution: Inverting the Inherence of Convolution for Visual Recognition](https://mlsanity.com/paper/2103.06255/) (2021)
- [A Comprehensive Study of Deep Video Action Recognition](https://mlsanity.com/paper/2012.06567/) (2020)
- [ACTION-Net: Multipath Excitation for Action Recognition](https://mlsanity.com/paper/2103.07372/) (2021)
- [Seeing Voices and Hearing Faces: Cross-modal biometric matching](https://mlsanity.com/paper/1804.00326/) (2018)
- [Real-time Action Recognition with Enhanced Motion Vector CNNs](https://mlsanity.com/paper/1604.07669/) (2016)
- [DMC-Net: Generating Discriminative Motion Cues for Fast Compressed Video Action Recognition](https://mlsanity.com/paper/1901.03460/) (2019)

## Built on by (most cited first, 5 of 5)

- [CoPE-VideoLM: Leveraging Codec Primitives For Efficient Video Language Modeling](https://mlsanity.com/paper/2602.13191/) (2026)
- [AdaCodec: A Predictive Visual Code for Video MLLMs](https://mlsanity.com/paper/2606.02569/) (2026)
- [Towards Scalable Modeling of Compressed Videos for Efficient Action Recognition](https://mlsanity.com/paper/2503.13724/) (2025)
- [Advancing Compressed Video Action Recognition through Progressive Knowledge Distillation](https://mlsanity.com/paper/2407.02713/) (2024)
- [RESUME: Recurrent State Updates from Motion and Residual Signals for Efficient Video Language Modeling](https://mlsanity.com/paper/2609.39563/) (2026)

## Abstract (arXiv)

Most of existing video action recognition models ingest raw RGB frames. However, the raw video stream requires enormous storage and contains significant temporal redundancy. Video compression (e.g., H.264, MPEG-4) reduces superfluous information by representing the raw video stream using the concept of Group of Pictures (GOP). Each GOP is composed of the first I-frame (aka RGB image) followed by a number of P-frames, represented by motion vectors and residuals, which can be regarded and used as pre-extracted features. In this work, we 1) introduce sampling the input for the network from partially decoded videos based on the GOP-level, and 2) propose a plug-and-play mulTi-modal lEArning Module (TEAM) for training the network using information from I-frames and P-frames in an end-to-end manner. We demonstrate the superior performance of TEAM-Net compared to the baseline using RGB only. TEAM-Net also achieves the state-of-the-art performance in the area of video action recognition with partial decoding. Code is provided at https://github.com/villawang/TEAM-Net.