mlsanityData to 14 Oct 2026

Time-Equivariant Contrastive Video Representation Learning

arXiv 2112.03624 ↗First posted 7 Dec 2021cs.CVcited by 44

Not on the map. Why?

Abstract (arXiv)

We introduce a novel self-supervised contrastive learning method to learn representations from unlabelled videos. Existing approaches ignore the specifics of input distortions, e.g., by learning invariance to temporal transformations. Instead, we argue that video representation should preserve video dynamics and reflect temporal manipulations of the input. Therefore, we exploit novel constraints to build representations that are equivariant to temporal transformations and better capture video dynamics. In our method, relative temporal transformations between augmented clips of a video are encoded in a vector and contrasted with other transformation vectors. To support temporal equivariance learning, we additionally propose the self-supervised classification of two clips of a video into 1. overlapping 2. ordered, or 3. unordered. Our experiments show that time-equivariant representations achieve state-of-the-art results in video retrieval and action recognition benchmarks on UCF101, HMDB51, and Diving48.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Builds on 61 of its 78 references that are in the map. First posted on arXiv 7 Dec 2021.

Builds on

61 of 78 references.

References · 61

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  6. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  7. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  8. 2017The Kinetics Human Action Video Datasetcited by 1,909
  9. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  10. 2017Decoupled Weight Decay Regularizationcited by 1,831
  11. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  12. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
38 more
  1. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  2. 2016Group Equivariant Convolutional Networkscited by 1,165
  3. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,143
  4. 2016Colorful Image Colorizationcited by 1,114
  5. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  6. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  7. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 902
  8. 2017Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?cited by 551
  9. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  10. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 507
  11. 2020Spatiotemporal Contrastive Video Representation Learningcited by 372
  12. 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
  13. 2016Harmonic Networks: Deep Translation and Rotation Equivariancecited by 295
  14. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 267
  15. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 249
  16. 2014Understanding image representations by measuring their equivariance and equivalencecited by 243
  17. 2020Self-supervised Co-training for Video Representation Learningcited by 232
  18. 2020What Should Not Be Contrastive in Contrastive Learningcited by 226
  19. 2017Unsupervised Representation Learning by Sorting Sequencescited by 221
  20. 2016Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Predictioncited by 217
  21. 2019Local Aggregation for Unsupervised Learning of Visual Embeddingscited by 204
  22. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 188
  23. 2017Colorization as a Proxy Task for Visual Understandingcited by 175
  24. 2019Video Representation Learning by Dense Predictive Codingcited by 175
  25. 2020SpeedNet: Learning the Speediness in Videoscited by 171
  26. 2021VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examplescited by 154
  27. 2018Self-Supervised Video Representation Learning with Space-Time Cubic Puzzlescited by 149
  28. 2020Self-supervised Video Representation Learning by Pace Predictioncited by 139
  29. 2020Memory-augmented Dense Predictive Coding for Video Representation Learningcited by 137
  30. 2021TCLR: Temporal Contrastive Learning for Video Representationcited by 136
  31. 2019Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognitioncited by 117
  32. 2020Video Playback Rate Perception for Self-supervisedSpatio-Temporal Representation Learningcited by 103
  33. 2020Video Representation Learning by Recognizing Temporal Transformationscited by 83
  34. 2019Oops! Predicting Unintentional Action in Videocited by 82
  35. 2020Video Cloze Procedure for Self-Supervised Spatio-Temporal Learningcited by 81
  36. 2016Ambient Sound Provides Supervision for Visual Learningcited by 78
  37. 2019AET vs. AED: Unsupervised Representation Learning by Auto-Encoding Transformations rather than Datacited by 77
  38. 2018Self-Supervised Spatiotemporal Feature Learning via Video Rotation Predictioncited by 71

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 20; 2023: 12; 2024: 7; 2025: 2; 2026: 3.

44 papers.

Sort:

By area:

  1. 2023Test of Time: Instilling Video-Language Models with a Sense of Timecited by 51
  2. 2022SPAct: Self-supervised Privacy Preservation for Action Recognitioncited by 47
  3. 2023SkeletonMAE: Graph-based Masked Autoencoder for Skeleton Sequence Pre-trainingcited by 41
  4. 2022Probabilistic Representations for Video Contrastive Learningcited by 34
  5. 2023Disentangling Spatial and Temporal Learning for Efficient Image-to-Video Transfer Learningcited by 32
  6. 2022Unsupervised Pre-training for Temporal Action Localization Taskscited by 31
  7. 2024Universal Time-Series Representation Learning: A Surveycited by 30
  8. 2022Static and Dynamic Concepts for Self-supervised Video Representation Learningcited by 23
  9. 2022Local Spatiotemporal Representation Learning for Longitudinally-consistent Neuroimage Analysiscited by 19
  10. 2023Audio-Visual Contrastive Learning with Temporal Self-Supervisioncited by 18
  11. 2023TimeBalance: Temporally-Invariant and Temporally-Distinctive Video Representations for Semi-Supervised Action Recognitioncited by 16
  12. 2022Learning from Untrimmed Videos: Self-Supervised Video Representation Learning with Hierarchical Consistencycited by 15
32 more
  1. 2023Tubelet-Contrastive Self-Supervision for Video-Efficient Generalizationcited by 12
  2. 2022Motion Sensitive Contrastive Learning for Self-supervised Video Representationcited by 11
  3. 2023No More Shortcuts: Realizing the Potential of Temporal Self-Supervisioncited by 11
  4. 2022Controllable Augmentations for Video Representation Learningcited by 10
  5. 2022TransRank: Self-supervised Video Representation Learning via Ranking-based Transformation Recognitioncited by 10
  6. 2023Language-based Action Concept Spaces Improve Video Self-Supervised Learningcited by 10
  7. 2024Self-supervised visual learning from interactions with objectscited by 10
  8. 2022Equivariant Self-Supervision for Musical Tempo Estimationcited by 9
  9. 2022Self-supervised Learning for Unintentional Action Predictioncited by 7
  10. 2022Learning Transferable Spatiotemporal Representations from Natural Script Knowledgecited by 6
  11. 2022Visually-aware Acoustic Event Detection using Heterogeneous Graphscited by 4
  12. 2024LocoMotion: Learning Motion-Focused Video-Language Representationscited by 4
  13. 2023Improving Equivariance in State-of-the-Art Supervised Depth and Normal Predictorscited by 3
  14. 2024Unfolding Videos Dynamics via Taylor Expansioncited by 3
  15. 2025Cloning Deterministic Worlds: The Critical Role of Latent Geometry in Long-Horizon World Modelscited by 3
  16. 2022CODiT: Conformal Out-of-Distribution Detection in Time-Series Datacited by 2
  17. 2022Spatio-Temporal Crop Aggregation for Video Representation Learningcited by 2
  18. 2022MoQuad: Motion-focused Quadruple Construction for Video Contrastive Learningcited by 2
  19. 2022Similarity Contrastive Estimation for Image and Video Soft Contrastive Self-Supervised Learningcited by 2
  20. 2022Video-ReTime: Learning Temporally Varying Speediness for Time Remappingcited by 1
  21. 2022GOCA: Guided Online Cluster Assignment for Self-Supervised Video Representation Learningcited by 1
  22. 2023CIPER: Combining Invariant and Equivariant Representations Using Contrastive and Predictive Learningcited by 1
  23. 2023Egocentric RGB+Depth Action Recognition in Industry-Like Settingscited by 1
  24. 2024Learning Temporally Equivariance for Degenerative Disease Progression in OCT by Predicting Future Representationscited by 1
  25. 2024Sync from the Sea: Retrieving Alignable Videos from Large-Scale Datasetscited by 1
  26. 2025Privacy Beyond Pixels: Latent Anonymization for Privacy-Preserving Video Understandingcited by 1
  27. 2022iBoot: Image-bootstrapped Self-Supervised Video Representation Learningcited by 0
  28. 2023Nearest-Neighbor Inter-Intra Contrastive Learning from Unlabeled Videoscited by 0
  29. 2024Collaboratively Self-supervised Video Representation Learning for Action Recognitioncited by 0
  30. 2026Focus-to-Perceive Representation Learning: A Cognition-Inspired Hierarchical Framework for Endoscopic Video Analysiscited by 0
  31. 2026ViTAL-X: Video-Text Alignment with Cross-Modal Temporal Editscited by 0
  32. 2026The Shape of Time: Video-Token Contrast for Temporal Understanding in VideoLMscited by 0