mlsanityData to 14 Oct 2026

A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning

arXiv 2104.14558 ↗First posted 29 Apr 2021cs.CV cs.AI cs.LGcited by 214

Not on the map. Why?

Abstract (arXiv)

We present a large-scale study on unsupervised spatiotemporal representation learning from videos. With a unified perspective on four recent image-based frameworks, we study a simple objective that can easily generalize all these methods to space-time. Our objective encourages temporally-persistent features in the same video, and in spite of its simplicity, it works surprisingly well across: (i) different unsupervised frameworks, (ii) pre-training datasets, (iii) downstream datasets, and (iv) backbone architectures. We draw a series of intriguing observations from this study, e.g., we discover that encouraging long-spanned persistency can be effective even if the timespan is 60 seconds. In addition to state-of-the-art results in multiple benchmarks, we report a few promising cases in which unsupervised pre-training can outperform its supervised counterpart. Code is made available at https://github.com/facebookresearch/SlowFast

Key facts. Cited by 214 papers in the map (data through 14 Oct 2026). Builds on 75 of its 96 references that are in the map. First posted on arXiv 29 Apr 2021.

Builds on

75 of 96 references.

References · 75

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  8. 2017Mask R-CNNcited by 7,277
  9. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  10. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  11. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  12. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
38 more
  1. 2014Going Deeper with Convolutionscited by 4,241
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  3. 2015Fast R-CNNcited by 2,933
  4. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  5. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  6. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  7. 2017The Kinetics Human Action Video Datasetcited by 1,909
  8. 2018SlowFast Networks for Video Recognitioncited by 1,867
  9. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  10. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  11. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  12. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  13. 2019Contrastive Multiview Codingcited by 1,301
  14. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  15. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,143
  16. 2016Colorful Image Colorizationcited by 1,114
  17. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  18. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  19. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  20. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  21. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
  22. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 537
  23. 2017Large Batch Training of Convolutional Networkscited by 535
  24. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
  25. 2017Look, Listen and Learncited by 458
  26. 2018Invariant Information Clustering for Unsupervised Image Classification and Segmentationcited by 455
  27. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
  28. 2015Unsupervised Learning of Video Representations using LSTMscited by 435
  29. 2020Spatiotemporal Contrastive Video Representation Learningcited by 372
  30. 2018A Short Note about Kinetics-600cited by 333
  31. 2019A Short Note on the Kinetics-700 Human Action Datasetcited by 310
  32. 2015Deep multi-scale video prediction beyond mean square errorcited by 300
  33. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  34. 2019Learning Correspondence from the Cycle-Consistency of Timecited by 269
  35. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 267
  36. 2017Objects that Soundcited by 256
  37. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 249
  38. 2020Self-supervised Co-training for Video Representation Learningcited by 232

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 31; 2022: 71; 2023: 48; 2024: 35; 2025: 22; 2026: 7.

214 papers.

Sort:

By area:

  1. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  2. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  3. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 531
  4. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
  5. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 429
  6. 2023A Cookbook of Self-Supervised Learningcited by 250
  7. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 207
  8. 2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learningcited by 193
  9. 2022Frozen CLIP Models are Efficient Video Learnerscited by 182
  10. 2021BEVT: BERT Pretraining of Video Transformerscited by 169
  11. 2024Streaming Long Video Understanding with Large Language Modelscited by 167
  12. 2023Meta-Transformer: A Unified Framework for Multimodal Learningcited by 136
38 more
  1. 2022Leveraging Real Talking Faces via Self-Supervision for Robust Forgery Detectioncited by 130
  2. 2024HourVideo: 1-Hour Video-Language Understandingcited by 113
  3. 2023Selective Structured State-Spaces for Long-Form Video Understandingcited by 105
  4. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 105
  5. 2022Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learningcited by 90
  6. 2022Learning To Recognize Procedural Activities with Distant Supervisioncited by 85
  7. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  8. 2023Siamese Masked Autoencoderscited by 76
  9. 2022A Comprehensive Survey on Deep Clustering: Taxonomy, Challenges, and Future Directionscited by 73
  10. 2023Diffusion Models as Masked Autoencoderscited by 71
  11. 2022SVFormer: Semi-supervised Video Transformer for Action Recognitioncited by 68
  12. 2021Self-supervised Video Transformercited by 64
  13. 2022Video Transformers: A Surveycited by 63
  14. 2023SignBERT+: Hand-model-aware Self-supervised Pre-training for Sign Language Understandingcited by 59
  15. 2023MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised Learning of Motion and Content Featurescited by 57
  16. 2024Koala: Key frame-conditioned long video-LLMcited by 57
  17. 2021VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learningcited by 56
  18. 2022Unsupervised Point Cloud Representation Learning with Deep Neural Networks: A Surveycited by 54
  19. 2022Semi-Supervised and Unsupervised Deep Visual Learning: A Surveycited by 52
  20. 2021Contrast and Mix: Temporal Contrastive Video Domain Adaptation with Background Mixingcited by 49
  21. 2022Dissecting Self-Supervised Learning Methods for Surgical Computer Visioncited by 48
  22. 2022SPAct: Self-supervised Privacy Preservation for Action Recognitioncited by 47
  23. 2021Motion-aware Contrastive Video Representation Learning via Foreground-background Mergingcited by 45
  24. 2024DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Controlcited by 44
  25. 2021Long-Short Temporal Contrastive Learning of Video Transformerscited by 42
  26. 2021Video Contrastive Learning with Global Contextcited by 41
  27. 2022Masked Motion Encoding for Self-Supervised Video Representation Learningcited by 39
  28. 2022Ranking Info Noise Contrastive Estimation: Boosting Contrastive Learning via Ranked Positivescited by 37
  29. 2023Time Does Tell: Self-Supervised Time-Tuning of Dense Image Representationscited by 35
  30. 2022Probabilistic Representations for Video Contrastive Learningcited by 34
  31. 2021PyTorchVideo: A Deep Learning Library for Video Understandingcited by 32
  32. 2022AssistQ: Affordance-centric Question-driven Task Completion for Egocentric Assistantcited by 32
  33. 2023Disentangling Spatial and Temporal Learning for Efficient Image-to-Video Transfer Learningcited by 32
  34. 2022Self-Supervised Predictive Learning: A Negative-Free Method for Sound Source Localization in Visual Scenescited by 31
  35. 2022XKD: Cross-modal Knowledge Distillation with Domain Alignment for Video Representation Learningcited by 31
  36. 2024Scaling 4D Representationscited by 30
  37. 2025AToken: A Unified Tokenizer for Visioncited by 30
  38. 2021Cross-Model Pseudo-Labeling for Semi-Supervised Action Recognitioncited by 29

50 most cited of 214 All 214 (JSON)