# AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition

arXiv: https://arxiv.org/abs/2102.05775 · HTML: https://mlsanity.com/paper/2102.05775/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 64 references that are in the map.
- First posted on arXiv 10 Feb 2021.

## Builds on (most cited first, 30 of 51)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [Adaptive Computation Time for Recurrent Neural Networks](https://mlsanity.com/paper/1603.08983/) (2016)
- [Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://mlsanity.com/paper/1411.4389/) (2014)
- [GhostNet: More Features from Cheap Operations](https://mlsanity.com/paper/1911.11907/) (2019)
- [Dynamic Convolution: Attention over Convolution Kernels](https://mlsanity.com/paper/1912.03458/) (2019)
- [SkipNet: Learning Dynamic Routing in Convolutional Networks](https://mlsanity.com/paper/1711.09485/) (2017)
- [Temporal Relational Reasoning in Videos](https://mlsanity.com/paper/1711.08496/) (2017)
- [Video Classification with Channel-Separated Convolutional Networks](https://mlsanity.com/paper/1904.02811/) (2019)
- [CondConv: Conditionally Parameterized Convolutions for Efficient Inference](https://mlsanity.com/paper/1904.04971/) (2019)
- [Videos as Space-Time Region Graphs](https://mlsanity.com/paper/1806.01810/) (2018)
- [Moments in Time Dataset: one million videos for event understanding](https://mlsanity.com/paper/1801.03150/) (2018)
- [BlockDrop: Dynamic Inference Paths in Residual Networks](https://mlsanity.com/paper/1711.08393/) (2017)
- [SpotTune: Transfer Learning through Adaptive Fine-tuning](https://mlsanity.com/paper/1811.08737/) (2018)
- [Conditional Computation in Neural Networks for faster models](https://mlsanity.com/paper/1511.06297/) (2015)
- [TEA: Temporal Excitation and Aggregation for Action Recognition](https://mlsanity.com/paper/2004.01398/) (2020)

## Built on by (most cited first, 30 of 48)

- [IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformers](https://mlsanity.com/paper/2106.12620/) (2021)
- [Adaptive Focus for Efficient Video Recognition](https://mlsanity.com/paper/2105.03245/) (2021)
- [FrameExit: Conditional Early Exiting for Efficient Video Recognition](https://mlsanity.com/paper/2104.13400/) (2021)
- [Glance and Focus Networks for Dynamic Visual Recognition](https://mlsanity.com/paper/2201.03014/) (2022)
- [AdaFocus V2: End-to-End Training of Spatial Dynamic Networks for Video Recognition](https://mlsanity.com/paper/2112.14238/) (2021)
- [AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition](https://mlsanity.com/paper/2105.05165/) (2021)
- [Dynamic Network Quantization for Efficient Video Inference](https://mlsanity.com/paper/2108.10394/) (2021)
- [TAda! Temporally-Adaptive Convolutions for Video Understanding](https://mlsanity.com/paper/2110.06178/) (2021)
- [Dynamic Perceiver for Efficient Visual Recognition](https://mlsanity.com/paper/2306.11248/) (2023)
- [EgoDistill: Egocentric Head Motion Distillation for Efficient Video Understanding](https://mlsanity.com/paper/2301.02217/) (2023)
- [HaLP: Hallucinating Latent Positives for Skeleton-based Self-Supervised Learning of Actions](https://mlsanity.com/paper/2304.00387/) (2023)
- [OCSampler: Compressing Videos to One Clip with Single-step Sampling](https://mlsanity.com/paper/2201.04388/) (2022)
- [Can An Image Classifier Suffice For Action Recognition?](https://mlsanity.com/paper/2106.14104/) (2021)
- [Efficient Deep Visual and Inertial Odometry with Adaptive Visual Modality Selection](https://mlsanity.com/paper/2205.06187/) (2022)
- [AdaFocusV3: On Unified Spatial-temporal Dynamic Video Recognition](https://mlsanity.com/paper/2209.13465/) (2022)
- [VA-RED2: Video Adaptive Redundancy Reduction](https://mlsanity.com/paper/2102.07887/) (2021)
- [Gate-Shift-Fuse for Video Action Recognition](https://mlsanity.com/paper/2203.08897/) (2022)
- [NSNet: Non-saliency Suppression Sampler for Efficient Video Recognition](https://mlsanity.com/paper/2207.10388/) (2022)
- [Computation-efficient Deep Learning for Computer Vision: A Survey](https://mlsanity.com/paper/2308.13998/) (2023)
- [MutualNet: Adaptive ConvNet via Mutual Learning from Different Model Configurations](https://mlsanity.com/paper/2105.07085/) (2021)
- [Unsupervised Domain Adaptation for Video Transformers in Action Recognition](https://mlsanity.com/paper/2207.12842/) (2022)
- [AdaBrowse: Adaptive Video Browser for Efficient Continuous Sign Language Recognition](https://mlsanity.com/paper/2308.08327/) (2023)
- [Temporal Saliency Query Network for Efficient Video Recognition](https://mlsanity.com/paper/2207.10379/) (2022)
- [Uni-AdaFocus: Spatial-temporal Dynamic Computation for Video Recognition](https://mlsanity.com/paper/2412.11228/) (2024)
- [Rethinking Resolution in the Context of Efficient Video Recognition](https://mlsanity.com/paper/2209.12797/) (2022)
- [TVNet: A Novel Time Series Analysis Method Based on Dynamic Convolution and 3D-Variation](https://mlsanity.com/paper/2503.07674/) (2025)
- [Temporally-Adaptive Models for Efficient Video Understanding](https://mlsanity.com/paper/2308.05787/) (2023)
- [Higher Order Recurrent Space-Time Transformer for Video Action Prediction](https://mlsanity.com/paper/2104.08665/) (2021)
- [Look More but Care Less in Video Recognition](https://mlsanity.com/paper/2211.09992/) (2022)
- [Event Neural Networks](https://mlsanity.com/paper/2112.00891/) (2021)

## Abstract (arXiv)

Temporal modelling is the key for efficient video action recognition. While understanding temporal information can improve recognition accuracy for dynamic actions, removing temporal redundancy and reusing past features can significantly save computation leading to efficient action recognition. In this paper, we introduce an adaptive temporal fusion network, called AdaFuse, that dynamically fuses channels from current and past feature maps for strong temporal modelling. Specifically, the necessary information from the historical convolution feature maps is fused with current pruned feature maps with the goal of improving both recognition accuracy and efficiency. In addition, we use a skipping operation to further reduce the computation cost of action recognition. Extensive experiments on Something V1 & V2, Jester and Mini-Kinetics show that our approach can achieve about 40% computation savings with comparable accuracy to state-of-the-art methods. The project page can be found at https://mengyuest.github.io/AdaFuse/