mlsanityData to 7 Oct 2026

Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification

arXiv 1712.04851 ↗First posted 13 Dec 2017cs.CVcited by 521

Not on the map. Why?

Abstract (arXiv)

Despite the steady progress in video analysis led by the adoption of convolutional neural networks (CNNs), the relative improvement has been less drastic as that in 2D static image classification. Three main challenges exist including spatial (image) feature representation, temporal information representation, and model/computation complexity. It was recently shown by Carreira and Zisserman that 3D CNNs, inflated from 2D networks and pretrained on ImageNet, could be a promising way for spatial and temporal representation learning. However, as for model/computation complexity, 3D CNNs are much more expensive than 2D CNNs and prone to overfit. We seek a balance between speed and accuracy by building an effective and efficient video classification system through systematic exploration of critical network design choices. In particular, we show that it is possible to replace many of the 3D convolutions by low-cost 2D convolutions. Rather surprisingly, best result (in both speed and accuracy) is achieved when replacing the 3D convolutions at the bottom of the network, suggesting that temporal representation learning on high-level semantic features is more useful. Our conclusion generalizes to datasets with very different properties. When combined with several other cost-effective designs including separable spatial/temporal convolution and feature gating, our system results in an effective video classification system that that produces very competitive results on several action classification benchmarks (Kinetics, Something-something, UCF101 and HMDB), as well as two action detection (localization) benchmarks (JHMDB and UCF101-24).

Key facts. Cited by 521 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Dec 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 155; 2022: 129; 2023: 103; 2024: 75; 2025: 42; 2026: 17.

521 papers.

Sort:

By area:

  1. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  2. 2021ViViT: A Video Vision Transformercited by 1,610
  3. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  4. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  5. 2021Video Swin Transformercited by 1,090
  6. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  7. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  8. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  9. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  10. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  11. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  12. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 320
38 more
  1. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 317
  2. 2022Egocentric Video-Language Pretrainingcited by 231
  3. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  4. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  5. 2023Video Understanding with Large Language Models: A Surveycited by 203
  6. 2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learningcited by 192
  7. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  8. 2022Frozen CLIP Models are Efficient Video Learnerscited by 181
  9. 2021SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioningcited by 161
  10. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  11. 2022Multiview Transformers for Video Recognitioncited by 148
  12. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 147
  13. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  14. 2021TCLR: Temporal Contrastive Learning for Video Representationcited by 136
  15. 2021MoViNets: Mobile Video Networks for Efficient Video Recognitioncited by 135
  16. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  17. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 128
  18. 2021TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?cited by 127
  19. 2021Video Frame Interpolation Transformercited by 122
  20. 2022Two-Stream Network for Sign Language Recognition and Translationcited by 122
  21. 2021A Survey of Visual Transformerscited by 117
  22. 2021Broaden Your Views for Self-Supervised Video Learningcited by 108
  23. 2024Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understandingcited by 106
  24. 2022TS2-Net: Token Shift and Selection Transformer for Text-Video Retrievalcited by 104
  25. 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 104
  26. 2021T2VLAD: Global-Local Sequence Alignment for Text-Video Retrievalcited by 103
  27. 2021VLM: Task-agnostic Video-Language Model Pre-training for Video Understandingcited by 102
  28. 2021TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignmentcited by 102
  29. 2023TALL: Thumbnail Layout for Deepfake Video Detectioncited by 100
  30. 2021VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluationcited by 95
  31. 2024Synchformer: Efficient Synchronization from Sparse Cuescited by 94
  32. 2023Vita-CLIP: Video and text adaptive CLIP via Multimodal Promptingcited by 90
  33. 2023Multiple Physics Pretraining for Physical Surrogate Modelscited by 90
  34. 2021HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrievalcited by 87
  35. 2022VindLU: A Recipe for Effective Video-and-Language Pretrainingcited by 87
  36. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  37. 2021Rethinking Self-supervised Correspondence Learning: A Video Frame-level Similarity Perspectivecited by 81
  38. 2022A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translationcited by 80

50 most cited of 521 All 521 (JSON)