mlsanityData to 7 Oct 2026

Continual 3D Convolutional Neural Networks for Real-time Processing of Videos

arXiv 2106.00050 ↗First posted 31 May 2021cs.CV cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

We introduce Continual 3D Convolutional Neural Networks (Co3D CNNs), a new computational formulation of spatio-temporal 3D CNNs, in which videos are processed frame-by-frame rather than by clip. In online tasks demanding frame-wise predictions, Co3D CNNs dispense with the computational redundancies of regular 3D CNNs, namely the repeated convolutions over frames, which appear in overlapping clips. We show that Continual 3D CNNs can reuse preexisting 3D-CNN weights to reduce the per-prediction floating point operations (FLOPs) in proportion to the temporal receptive field while retaining similar memory requirements and accuracy. This is validated with multiple models on Kinetics-400 and Charades with remarkable results: CoX3D models attain state-of-the-art complexity/accuracy trade-offs on Kinetics-400 with 12.1-15.3x reductions of FLOPs and 2.3-3.8% improvements in accuracy compared to regular X3D models while reducing peak memory consumption by up to 48%. Moreover, we investigate the transient response of Co3D CNNs at start-up and perform extensive benchmarks of on-hardware processing characteristics for publicly available 3D CNNs.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 34 of its 49 references that are in the map. First posted on arXiv 31 May 2021.

Builds on

34 of 49 references.

References · 34

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  4. 2017Squeeze-and-Excitation Networkscited by 4,763
  5. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  6. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  7. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  8. 2017The Kinetics Human Action Video Datasetcited by 1,894
  9. 2018SlowFast Networks for Video Recognitioncited by 1,862
  10. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  11. 2021ViViT: A Video Vision Transformercited by 1,610
  12. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
22 more
  1. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  2. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  3. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  4. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  5. 2018Deep Visual Domain Adaptation: A Surveycited by 634
  6. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  7. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 483
  8. 2018A Short Note about Kinetics-600cited by 331
  9. 2021Video Transformer Networkcited by 271
  10. 2015Beyond Short Snippets: Deep Networks for Video Classificationcited by 233
  11. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 209
  12. 2015Compressing Neural Networks with the Hashing Trickcited by 150
  13. 2017Deep Learning with Low Precision by Half-wave Gaussian Quantizationcited by 115
  14. 2020Deep Visual Domain Adaptationcited by 91
  15. 2017Action Tubelet Detector for Spatio-Temporal Action Localizationcited by 77
  16. 2019You Only Watch Once: A Unified CNN Architecture for Real-Time Spatiotemporal Action Localizationcited by 58
  17. 2016Online Real-time Multiple Spatiotemporal Action Localisation and Predictioncited by 57
  18. 2017DeepCache: Principled Cache for Mobile Deep Visioncited by 57
  19. 2019Resource Efficient 3D Convolutional Neural Networkscited by 50
  20. 2018Partial Convolution based Paddingcited by 29
  21. 2018Massively Parallel Video Networkscited by 16
  22. 2020Dissected 3D CNNs: Temporal Skip Connections for Efficient Online Video Processingcited by 5

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 1; 2024: 2; 2025: 1.

9 papers.

  1. 2022Continual Transformers: Redundancy-Free Attention for Online Inferencecited by 12
  2. 2022Structured Pruning Adapterscited by 12
  3. 2022Efficient High-Resolution Deep Learning: A Surveycited by 9
  4. 2022Continual Spatio-Temporal Graph Convolutional Networkscited by 8
  5. 2022Continual Inference: A Library for Efficient Online Inference with Deep Neural Networks in PyTorchcited by 8
  6. 2024Continual Low-Rank Scaled Dot-product Attentioncited by 2
  7. 2023Efficient Online Processing with Deep Neural Networkscited by 0
  8. 2024PrAViC: Probabilistic Adaptation Framework for Real-Time Video Classificationcited by 0
  9. 2025DeepCoT: Deep Continual Transformers for Real-Time Inference on Data Streamscited by 0