mlsanityData to 14 Oct 2026

Self-supervised Video Transformer

arXiv 2112.01514 ↗First posted 2 Dec 2021cs.CVcited by 64

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose self-supervised training for video transformers using unlabeled video data. From a given video, we create local and global spatiotemporal views with varying spatial sizes and frame rates. Our self-supervised objective seeks to match the features of these different views representing the same video, to be invariant to spatiotemporal variations in actions. To the best of our knowledge, the proposed approach is the first to alleviate the dependency on negative samples or dedicated memory banks in Self-supervised Video Transformer (SVT). Further, owing to the flexibility of Transformer models, SVT supports slow-fast video processing within a single architecture using dynamically adjusted positional encoding and supports long-term relationship modeling along spatiotemporal dimensions. Our approach performs well on four action recognition benchmarks (Kinetics-400, UCF-101, HMDB-51, and SSv2) and converges faster with small batch sizes. Code: https://git.io/J1juJ

Key facts. Cited by 64 papers in the map (data through 14 Oct 2026). Builds on 78 of its 90 references that are in the map. First posted on arXiv 2 Dec 2021.

Builds on

78 of 90 references.

References · 78

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  10. 2020End-to-End Object Detection with Transformerscited by 7,414
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  12. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
38 more
  1. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  3. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  4. 2017Non-local Neural Networkscited by 2,293
  5. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  6. 2018SlowFast Networks for Video Recognitioncited by 1,867
  7. 2021Restormer: Efficient Transformer for High-Resolution Image Restorationcited by 1,627
  8. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  9. 2021ViViT: A Video Vision Transformercited by 1,620
  10. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  11. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  12. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  13. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  14. 2019Contrastive Multiview Codingcited by 1,301
  15. 2016Colorful Image Colorizationcited by 1,114
  16. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  17. 2021Video Swin Transformercited by 1,091
  18. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  19. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  20. 2021Transformers in Vision: A Surveycited by 1,013
  21. 2021Multiscale Vision Transformerscited by 897
  22. 2020What Makes for Good Views for Contrastive Learning?cited by 846
  23. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  24. 2015Unsupervised Deep Embedding for Clustering Analysiscited by 767
  25. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  26. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
  27. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 569
  28. 2019Self-labelling via simultaneous clustering and representation learningcited by 528
  29. 2021Intriguing Properties of Vision Transformerscited by 446
  30. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  31. 2015Unsupervised Learning of Video Representations using LSTMscited by 435
  32. 2016Generating Videos with Scene Dynamicscited by 376
  33. 2020Spatiotemporal Contrastive Video Representation Learningcited by 372
  34. 2020Keystroke Dynamics as Part of Lifeloggingcited by 334
  35. 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
  36. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 300
  37. 2015Deep multi-scale video prediction beyond mean square errorcited by 300
  38. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 267

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 16; 2023: 25; 2024: 11; 2025: 9; 2026: 3.

64 papers.

Sort:

By area:

  1. 2023Vita-CLIP: Video and text adaptive CLIP via Multimodal Promptingcited by 91
  2. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  3. 2023Foundation Model for Endoscopy Video Analysis via Large-scale Self-supervised Pre-traincited by 75
  4. 2022Video Transformers: A Surveycited by 63
  5. 2024LLaRA: Supercharging Robot Learning Data for Vision-Language Policycited by 52
  6. 2022SPAct: Self-supervised Privacy Preservation for Action Recognitioncited by 47
  7. 20233Mformer: Multi-order Multi-mode Transformer for Skeletal Action Recognitioncited by 40
  8. 2022Masked Motion Encoding for Self-Supervised Video Representation Learningcited by 39
  9. 2022How to Train Vision Transformer on Small-scale Datasets?cited by 35
  10. 2023Time Does Tell: Self-Supervised Time-Tuning of Dense Image Representationscited by 35
  11. 2022ResFormer: Scaling ViTs with Multi-Resolution Trainingcited by 34
  12. 2023Anatomical Invariance Modeling and Semantic Alignment for Self-supervised Learning in 3D Medical Image Analysiscited by 26
38 more
  1. 2024Understanding Long Videos with Multimodal Language Modelscited by 25
  2. 2022Does Self-supervised Learning Really Improve Reinforcement Learning from Pixels?cited by 24
  3. 2024Understanding Video Transformers via Universal Concept Discoverycited by 23
  4. 2022An Investigation into Whitening Loss for Self-supervised Learningcited by 21
  5. 2023SVFAP: Self-supervised Video Facial Affect Perceivercited by 18
  6. 2023Limited Data, Unlimited Potential: A Study on ViTs Augmented by Masked Autoencoderscited by 17
  7. 2023LAC: Latent Action Composition for Skeleton-based Action Segmentationcited by 15
  8. 2023SPARTAN: Self-supervised Spatiotemporal Transformers Approach to Group Activity Recognitioncited by 14
  9. 2023TVTSv2: Learning Out-of-the-box Spatiotemporal Visual Representations at Scalecited by 14
  10. 2024A Survey of Generative Techniques for Spatial-Temporal Data Miningcited by 14
  11. 2024SIGMA: Sinkhorn-Guided Masked Video Modelingcited by 14
  12. 2022Learning Viewpoint-Agnostic Visual Representations by Recovering Tokens in 3D Spacecited by 13
  13. 2023Tubelet-Contrastive Self-Supervision for Video-Efficient Generalizationcited by 12
  14. 2025Heterogeneous Skeleton-Based Action Representation Learningcited by 12
  15. 2023Self-Supervised Video Representation Learning via Latent Time Navigationcited by 11
  16. 2023No More Shortcuts: Realizing the Potential of Temporal Self-Supervisioncited by 11
  17. 2022It Takes Two: Masked Appearance-Motion Modeling for Self-supervised Video Transformer Pre-trainingcited by 10
  18. 2022TransVisDrone: Spatio-Temporal Transformer for Vision-based Drone-to-Drone Detection in Aerial Videoscited by 10
  19. 2023Language-based Action Concept Spaces Improve Video Self-Supervised Learningcited by 10
  20. 2023Modulate Your Spectrum in Self-Supervised Learningcited by 9
  21. 2025SMILE: Infusing Spatial and Motion Semantics in Masked Video Learningcited by 9
  22. 2022ViT-ReT: Vision and Recurrent Transformer Neural Networks for Human Activity Recognition in Videoscited by 7
  23. 2024Open-Vocabulary Temporal Action Localization using Multimodal Guidancecited by 7
  24. 2022Learning Transferable Spatiotemporal Representations from Natural Script Knowledgecited by 6
  25. 2023Self-Supervised Video Transformers for Isolated Sign Language Recognitioncited by 6
  26. 2023Offline-to-Online Knowledge Distillation for Video Instance Segmentationcited by 5
  27. 2023A Large-Scale Analysis on Self-Supervised Video Representation Learningcited by 5
  28. 2023REACT: Recognize Every Action Everywhere All At Oncecited by 5
  29. 2024ARVideo: Autoregressive Pretraining for Self-Supervised Video Representation Learningcited by 5
  30. 2023Video Action Recognition Collaborative Learning with Dynamics via PSO-ConvNet Transformercited by 4
  31. 2023CycleCL: Self-supervised Learning for Periodic Videoscited by 4
  32. 2022Self-supervised Video-centralised Transformer for Video Face Clusteringcited by 3
  33. 2023Attending Generalizability in Course of Deep Fake Detection by Exploring Multi-task Learningcited by 3
  34. 2025A Large-Scale Analysis on Contextual Self-Supervised Video Representation Learningcited by 3
  35. 2022Spatio-Temporal Crop Aggregation for Video Representation Learningcited by 2
  36. 2024Multi-Granularity Language-Guided Training for Multi-Object Trackingcited by 2
  37. 2025FRAME: Pre-Training Video Feature Representations via Anticipation and Memorycited by 2
  38. 2022Human Activity Recognition Using Cascaded Dual Attention CNN and Bi-Directional GRU Frameworkcited by 1

50 most cited of 64 All 64 (JSON)