mlsanityData to 7 Oct 2026

FrameExit: Conditional Early Exiting for Efficient Video Recognition

arXiv 2104.13400 ↗First posted 27 Apr 2021cs.CV cs.LGcited by 59

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose a conditional early exiting framework for efficient video recognition. While existing works focus on selecting a subset of salient frames to reduce the computation costs, we propose to use a simple sampling strategy combined with conditional early exiting to enable efficient recognition. Our model automatically learns to process fewer frames for simpler videos and more frames for complex ones. To achieve this, we employ a cascade of gating modules to automatically determine the earliest point in processing where an inference is sufficiently reliable. We generate on-the-fly supervision signals to the gates to provide a dynamic trade-off between accuracy and computational cost. Our proposed model outperforms competing methods on three large-scale video benchmarks. In particular, on ActivityNet1.3 and mini-kinetics, we outperform the state-of-the-art efficient video recognition methods with 1.3× and 2.1× less GFLOPs, respectively. Additionally, our method sets a new state of the art for efficient video understanding on the HVU benchmark.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Builds on 46 of its 63 references that are in the map. First posted on arXiv 27 Apr 2021.

Builds on

46 of 63 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  5. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  6. 2017The Kinetics Human Action Video Datasetcited by 1,894
  7. 2018SlowFast Networks for Video Recognitioncited by 1,862
  8. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  9. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  10. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  11. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  12. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
34 more
  1. 2017Learning Spatio-Temporal Representation with Pseudo-3D Residual Networkscited by 386
  2. 2014Deeply-Supervised Netscited by 364
  3. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
  4. 2017Temporal Relational Reasoning in Videoscited by 324
  5. 2019Video Classification with Channel-Separated Convolutional Networkscited by 290
  6. 2015Beyond Short Snippets: Deep Networks for Video Classificationcited by 233
  7. 2017BlockDrop: Dynamic Inference Paths in Residual Networkscited by 185
  8. 2017Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognitioncited by 178
  9. 2019Listen to Look: Action Recognition by Previewing Audiocited by 163
  10. 2017Convolutional Networks with Adaptive Inference Graphscited by 147
  11. 2020Resolution Adaptive Networks for Efficient Inferencecited by 146
  12. 2018AdaFrame: Adaptive Frame Selection for Fast Video Recognitioncited by 134
  13. 2019SCSampler: Sampling Salient Clips from Video for Efficient Action Recognitioncited by 133
  14. 2018Dynamic Channel Pruning: Feature Boosting and Suppressioncited by 129
  15. 2016Spatially Adaptive Computation Time for Residual Networkscited by 118
  16. 2018ECO: Efficient Convolutional Network for Online Video Understandingcited by 115
  17. 2020Conditional Channel Gated Networks for Task-Aware Continual Learningcited by 115
  18. 2015End-to-end Learning of Action Detection from Frame Glimpses in Videoscited by 101
  19. 2019Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inferencecited by 91
  20. 2020AR-Net: Adaptive Frame Resolution for Efficient Action Recognitioncited by 88
  21. 2019Multi-Agent Reinforcement Learning Based Frame Sampling for Effective Untrimmed Video Recognitioncited by 66
  22. 2019LiteEval: A Coarse-to-Fine Framework for Resource Efficient Video Recognitioncited by 65
  23. 2019More Is Less: Learning Efficient Video Representations by Big-Little Network and Depthwise Temporal Aggregationcited by 64
  24. 2019Resource Efficient 3D Convolutional Neural Networkscited by 50
  25. 2017Skip RNN: Learning to Skip State Updates in Recurrent Neural Networkscited by 49
  26. 2021AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognitioncited by 48
  27. 2019Batch-Shaping for Learning Conditional Channel Gated Networkscited by 42
  28. 2017ConvNet Architecture Search for Spatiotemporal Feature Learningcited by 40
  29. 2020Dynamic Sampling Networks for Efficient Action Recognition in Videoscited by 34
  30. 2015Rank Pooling for Action Recognitioncited by 29
  31. 2019Tiny Video Networkscited by 27
  32. 2018Video Time: Properties, Encoders and Evaluationcited by 19
  33. 2020TimeGate: Conditional Gating of Segments in Long-range Activitiescited by 7
  34. 2020Exploring Sparsity in Image Super-Resolution for Efficient Inferencecited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 15; 2023: 13; 2024: 13; 2025: 9; 2026: 6.

59 papers.

Sort:

By area:

  1. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  2. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 146
  3. 2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Executioncited by 116
  4. 2025Flash-VStream: Efficient Real-Time Understanding for Long Video Streamscited by 78
  5. 2022FERV39k: A Large-Scale Multi-Scene Dataset for Facial Expression Recognition in Videoscited by 69
  6. 2021AdaFocus V2: End-to-End Training of Spatial Dynamic Networks for Video Recognitioncited by 37
  7. 2023EgoDistill: Egocentric Head Motion Distillation for Efficient Video Understandingcited by 32
  8. 2022Predictive Exit: Prediction of Fine-Grained Early Exits for Computation- and Energy-Efficient Inferencecited by 24
  9. 2022OCSampler: Compressing Videos to One Clip with Single-step Samplingcited by 22
  10. 2022AdaFocusV3: On Unified Spatial-temporal Dynamic Video Recognitioncited by 18
  11. 2022Cloud-Device Collaborative Adaptation to Continual Changing Environments in the Real-worldcited by 16
  12. 2022NSNet: Non-saliency Suppression Sampler for Efficient Video Recognitioncited by 15
38 more
  1. 2022SMAUG: Sparse Masked Autoencoder for Efficient Video-Language Pre-trainingcited by 14
  2. 2023Computation-efficient Deep Learning for Computer Vision: A Surveycited by 14
  3. 2022Active Acquisition for Multimodal Temporal Data: A Challenging Decision-Making Taskcited by 11
  4. 2024Early Time Classification with Accumulated Accuracy Gap Controlcited by 11
  5. 2023AdaBrowse: Adaptive Video Browser for Efficient Continuous Sign Language Recognitioncited by 10
  6. 2024Conditional computation in neural networks: principles and research trendscited by 10
  7. 2025Towards Understanding How Knowledge Evolves in Large Vision-Language Modelscited by 10
  8. 2022Task-adaptive Spatial-Temporal Video Sampler for Few-shot Action Recognitioncited by 9
  9. 2022Temporal Saliency Query Network for Efficient Video Recognitioncited by 9
  10. 2024Uni-AdaFocus: Spatial-temporal Dynamic Computation for Video Recognitioncited by 9
  11. 2022Rethinking Resolution in the Context of Efficient Video Recognitioncited by 8
  12. 2022Edge Video Analytics: A Survey on Applications, Systems and Enabling Techniquescited by 7
  13. 2023PMI Sampler: Patch Similarity Guided Frame Selection for Aerial Action Recognitioncited by 7
  14. 2023Few-shot Action Recognition via Intra- and Inter-Video Information Maximizationcited by 7
  15. 2023Revisiting Kernel Temporal Segmentation as an Adaptive Tokenizer for Long-form Video Understandingcited by 7
  16. 2024Exploring Dynamic Transformer for Efficient Object Trackingcited by 7
  17. 2022Fluid Batching: Exit-Aware Preemptive Serving of Early-Exit Neural Networks on Edge NPUscited by 6
  18. 2021Event Neural Networkscited by 5
  19. 2023AccDecoder: Accelerated Decoding for Neural-enhanced Video Analyticscited by 5
  20. 2022ViGAT: Bottom-up event recognition and explanation in video using factorized graph attention networkcited by 4
  21. 2023View while Moving: Efficient Video Recognition in Long-untrimmed Videoscited by 4
  22. 2023Audio-Visual Glance Network for Efficient Video Recognitioncited by 4
  23. 2024UR2M: Uncertainty and Resource-Aware Event Detection on Microcontrollerscited by 4
  24. 2025EgoAdapt: Adaptive Multisensory Distillation and Policy Learning for Efficient Egocentric Perceptioncited by 4
  25. 2021The Power of Log-Sum-Exp: Sequential Density Ratio Matrix Estimation for Speed-Accuracy Optimizationcited by 3
  26. 2023DNArch: Learning Convolutional Neural Architectures by Backpropagationcited by 3
  27. 2025E4: Energy-Efficient DNN Inference for Edge Video Analytics Via Early-Exit and DVFScited by 2
  28. 2026AdaSpot: Spend Resolution Where It Matters for Precise Event Spottingcited by 2
  29. 2022QuickNets: Saving Training and Preventing Overconfidence in Early-Exit Neural Architecturescited by 1
  30. 2023Gated-ViGAT: Efficient Bottom-Up Event Recognition and Explanation Using a New Frame Selection Policy and Gating Mechanismcited by 1
  31. 2024Scalable Frame Sampling for Video Classification: A Semi-Optimal Policy Approach with Reduced Search Spacecited by 1
  32. 2024EdgeOAR: Real-time Online Action Recognition On Edge Devicescited by 1
  33. 2026MistExit: Learning to Exit for Early Mistake Detection in Procedural Videoscited by 1
  34. 2026Shot-Aware Frame Sampling for Video Understandingcited by 1
  35. 2023Context-Aware Iteration Policy Network for Efficient Optical Flow Estimationcited by 0
  36. 2024HaltingVT: Adaptive Token Halting Transformer for Efficient Video Recognitioncited by 0
  37. 2024Successive Refinement in Large-Scale Computation: Advancing Model Inference Applicationscited by 0
  38. 2024Learning Motion Blur Robust Vision Transformers for Real-Time UAV Trackingcited by 0

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021Temporal Early Exits for Efficient Video Object Detection
  2. 2022Frozen CLIP Models are Efficient Video Learners
  3. 2026Towards Data-Efficient Video Pre-training with Frozen Image Foundation Models
  4. 2025MoCLIP-Lite: Efficient Video Recognition by Fusing CLIP with Motion Vectors
  5. 2025LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervision
  6. 2025FrameOracle: Learning What to See and How Much to See in Videos
  7. 2022Revisiting Classifier: Transferring Vision-Language Models for Video Recognition
  8. 2025ResidualViT for Efficient Temporally Dense Video Encoding