mlsanityData to 7 Oct 2026

Is 3D Convolution with 5D Tensors Really Necessary for Video Analysis?

arXiv 2407.16514 ↗First posted 23 Jul 2024cs.CV cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, we present a comprehensive study and propose several novel techniques for implementing 3D convolutional blocks using 2D and/or 1D convolutions with only 4D and/or 3D tensors. Our motivation is that 3D convolutions with 5D tensors are computationally very expensive and they may not be supported by some of the edge devices used in real-time applications such as robots. The existing approaches mitigate this by splitting the 3D kernels into spatial and temporal domains, but they still use 3D convolutions with 5D tensors in their implementations. We resolve this issue by introducing some appropriate 4D/3D tensor reshaping as well as new combination techniques for spatial and temporal splits. The proposed implementation methods show significant improvement both in terms of efficiency and accuracy. The experimental results confirm that the proposed spatio-temporal processing structure outperforms the original model in terms of speed and accuracy using only 4D tensors with fewer parameters.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 34 of its 42 references that are in the map. First posted on arXiv 23 Jul 2024.

Builds on

34 of 42 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2014Going Deeper with Convolutionscited by 4,231
  5. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  6. 2017Non-local Neural Networkscited by 2,290
  7. 2017The Kinetics Human Action Video Datasetcited by 1,894
  8. 2018SlowFast Networks for Video Recognitioncited by 1,862
  9. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
  10. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  11. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  12. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
22 more
  1. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  2. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  3. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  4. 2017Learning Spatio-Temporal Representation with Pseudo-3D Residual Networkscited by 386
  5. 2019Video Classification with Channel-Separated Convolutional Networkscited by 290
  6. 2016A guide to convolution arithmetic for deep learningcited by 182
  7. 2020TEA: Temporal Excitation and Aggregation for Action Recognitioncited by 165
  8. 2020Temporal Pyramid Network for Action Recognitioncited by 163
  9. 2018ECO: Efficient Convolutional Network for Online Video Understandingcited by 115
  10. 2016Long-term Temporal Convolutions for Action Recognitioncited by 84
  11. 2019AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architecturescited by 46
  12. 2020Temporal Interlacing Networkcited by 45
  13. 2017ConvNet Architecture Search for Spatiotemporal Feature Learningcited by 40
  14. 2020V4D:4D Convolutional Neural Networks for Video-level Representation Learningcited by 33
  15. 2014Flattened Convolutional Neural Networks for Feedforward Accelerationcited by 20
  16. 2023GQKVA: Efficient Pre-training of Transformers by Grouping Queries, Keys, and Valuescited by 9
  17. 2023Training Acceleration of Low-Rank Decomposed Networks using Sequential Freezing and Rank Quantizationcited by 8
  18. 2023Speeding up Resnet Architecture with Layers Targeted Low Rank Decompositioncited by 8
  19. 2024SkipViT: Speeding Up Vision Transformers with a Token-Level Skip Connectioncited by 5
  20. 2023SwiftLearn: A Data-Efficient Training Method of Deep Learning Models using Importance Samplingcited by 3
  21. 2024Single Parent Family: A Spectrum of Family Members from a Single Pre-Trained Foundation Modelcited by 3
  22. 2018Rank-1 Convolutional Neural Networkcited by 1

Built on by

Not cited yet.