# 2D or not 2D? Adaptive 3D Convolution Selection for Efficient Video Recognition

arXiv: https://arxiv.org/abs/2012.14950 · HTML: https://mlsanity.com/paper/2012.14950/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Dec 2020.

## Built on by (most cited first, 22 of 22)

- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [AdaViT: Adaptive Vision Transformers for Efficient Image Recognition](https://mlsanity.com/paper/2111.15668/) (2021)
- [Adaptive Focus for Efficient Video Recognition](https://mlsanity.com/paper/2105.03245/) (2021)
- [AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruning](https://mlsanity.com/paper/2412.03248/) (2024)
- [Towards Scalable Neural Representation for Diverse Videos](https://mlsanity.com/paper/2303.14124/) (2023)
- [MAR: Masked Autoencoders for Efficient Action Recognition](https://mlsanity.com/paper/2207.11660/) (2022)
- [OCSampler: Compressing Videos to One Clip with Single-step Sampling](https://mlsanity.com/paper/2201.04388/) (2022)
- [NSNet: Non-saliency Suppression Sampler for Efficient Video Recognition](https://mlsanity.com/paper/2207.10388/) (2022)
- [Multi-event Video-Text Retrieval](https://mlsanity.com/paper/2308.11551/) (2023)
- [Computation-efficient Deep Learning for Computer Vision: A Survey](https://mlsanity.com/paper/2308.13998/) (2023)
- [Learning to Inference Adaptively for Multimodal Large Language Models](https://mlsanity.com/paper/2503.10905/) (2025)
- [ERA: Expert Retrieval and Assembly for Early Action Prediction](https://mlsanity.com/paper/2207.09675/) (2022)
- [Temporal Saliency Query Network for Efficient Video Recognition](https://mlsanity.com/paper/2207.10379/) (2022)
- [System-status-aware Adaptive Network for Online Streaming Video Understanding](https://mlsanity.com/paper/2303.15742/) (2023)
- [GradMDM: Adversarial Attack on Dynamic Networks](https://mlsanity.com/paper/2304.06724/) (2023)
- [ViGAT: Bottom-up event recognition and explanation in video using factorized graph attention network](https://mlsanity.com/paper/2207.09927/) (2022)
- [View while Moving: Efficient Video Recognition in Long-untrimmed Videos](https://mlsanity.com/paper/2308.04834/) (2023)
- [E^2TAD: An Energy-Efficient Tracking-based Action Detector](https://mlsanity.com/paper/2204.04416/) (2022)
- [Gated-ViGAT: Efficient Bottom-Up Event Recognition and Explanation Using a New Frame Selection Policy and Gating Mechanism](https://mlsanity.com/paper/2301.07565/) (2023)
- [Scalable Frame Sampling for Video Classification: A Semi-Optimal Policy Approach with Reduced Search Space](https://mlsanity.com/paper/2409.05260/) (2024)
- [Soften the Mask: Adaptive Temporal Soft Mask for Efficient Dynamic Facial Expression Recognition](https://mlsanity.com/paper/2502.21004/) (2025)
- [DANCE: Dynamic 3D CNN Pruning: Joint Frame, Channel, and Feature Adaptation for Energy Efficiency on the Edge](https://mlsanity.com/paper/2603.17275/) (2026)

## Abstract (arXiv)

3D convolutional networks are prevalent for video recognition. While achieving excellent recognition performance on standard benchmarks, they operate on a sequence of frames with 3D convolutions and thus are computationally demanding. Exploiting large variations among different videos, we introduce Ada3D, a conditional computation framework that learns instance-specific 3D usage policies to determine frames and convolution layers to be used in a 3D network. These policies are derived with a two-head lightweight selection network conditioned on each input video clip. Then, only frames and convolutions that are selected by the selection network are used in the 3D model to generate predictions. The selection network is optimized with policy gradient methods to maximize a reward that encourages making correct predictions with limited computation. We conduct experiments on three video recognition benchmarks and demonstrate that our method achieves similar accuracies to state-of-the-art 3D models while requiring 20%-50% less computation across different datasets. We also show that learned policies are transferable and Ada3D is compatible to different backbones and modern clip selection approaches. Our qualitative analysis indicates that our method allocates fewer 3D convolutions and frames for "static" inputs, yet uses more for motion-intensive clips.