mlsanityData to 7 Oct 2026

Static and Dynamic Concepts for Self-supervised Video Representation Learning

arXiv 2207.12795 ↗First posted 26 Jul 2022cs.CV cs.LGcited by 23

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose a novel learning scheme for self-supervised video representation learning. Motivated by how humans understand videos, we propose to first learn general visual concepts then attend to discriminative local areas for video understanding. Specifically, we utilize static frame and frame difference to help decouple static and dynamic concepts, and respectively align the concept distributions in latent space. We add diversity and fidelity regularizations to guarantee that we learn a compact set of meaningful concepts. Then we employ a cross-attention mechanism to aggregate detailed local features of different concepts, and filter out redundant concepts with low activations to perform local concept contrast. Extensive experiments demonstrate that our method distills meaningful static and dynamic concepts to guide video understanding, and obtains state-of-the-art results on UCF-101, HMDB-51, and Diving-48.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 74 of its 85 references that are in the map. First posted on arXiv 26 Jul 2022.

Builds on

74 of 85 references.

References · 74

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  7. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  8. 2017Non-local Neural Networkscited by 2,290
  9. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  10. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  11. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  12. 2019Contrastive Multiview Codingcited by 1,297
38 more
  1. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  2. 2020Concept Bottleneck Modelscited by 1,022
  3. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  4. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  5. 2020Dense Contrastive Learning for Self-Supervised Visual Pre-Trainingcited by 512
  6. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  7. 2018Towards Robust Interpretability with Self-Explaining Neural Networkscited by 505
  8. 2020Spatiotemporal Contrastive Video Representation Learningcited by 372
  9. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 299
  10. 2020Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learningcited by 297
  11. 2020Whitening for Self-Supervised Representation Learningcited by 278
  12. 2019Learning Correspondence from the Cycle-Consistency of Timecited by 269
  13. 2020Space-Time Correspondence as a Contrastive Random Walkcited by 239
  14. 2020Self-supervised Co-training for Video Representation Learningcited by 232
  15. 2020Concept Whitening for Interpretable Image Recognitioncited by 225
  16. 2021A Large-Scale Study on Unsupervised Spatiotemporal Representation Learningcited by 214
  17. 2019Video Representation Learning by Dense Predictive Codingcited by 175
  18. 2020SpeedNet: Learning the Speediness in Videoscited by 171
  19. 2021VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examplescited by 154
  20. 2018Self-Supervised Video Representation Learning with Space-Time Cubic Puzzlescited by 149
  21. 2020Self-supervised Video Representation Learning by Pace Predictioncited by 139
  22. 2020Memory-augmented Dense Predictive Coding for Video Representation Learningcited by 137
  23. 2021TCLR: Temporal Contrastive Learning for Video Representationcited by 136
  24. 2022Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generationcited by 119
  25. 2020Multiple Sound Sources Localization from Coarse to Finecited by 118
  26. 2021Broaden Your Views for Self-Supervised Video Learningcited by 108
  27. 2022Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generationcited by 106
  28. 2020Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matchingcited by 104
  29. 2020Video Playback Rate Perception for Self-supervisedSpatio-Temporal Representation Learningcited by 103
  30. 2020Labelling unlabelled videos from scratch with multi-modal self-supervisioncited by 99
  31. 2019Self-supervised Spatio-temporal Representation Learning for Videos by Predicting Motion and Appearance Statisticscited by 92
  32. 2020RSPNet: Relative Speed Perception for Unsupervised Video Representation Learningcited by 83
  33. 2020Video Representation Learning by Recognizing Temporal Transformationscited by 82
  34. 2019Joint-task Self-supervised Learning for Temporal Correspondencecited by 80
  35. 2020Video Cloze Procedure for Self-Supervised Spatio-Temporal Learningcited by 80
  36. 2022Concept Bottleneck Model with Additional Unsupervised Conceptscited by 79
  37. 2020Evolving Losses for Unsupervised Video Representation Learningcited by 74
  38. 2018Learning Image Representations by Completing Damaged Jigsaw Puzzlescited by 66

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 6; 2025: 5; 2026: 2.

23 papers.

Sort:

By area:

  1. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  2. 2024HourVideo: 1-Hour Video-Language Understandingcited by 110
  3. 2023Foundation Model for Endoscopy Video Analysis via Large-scale Self-supervised Pre-traincited by 75
  4. 2023Prune Spatio-temporal Tokens by Semantic-aware Temporal Accumulationcited by 23
  5. 2023Semantics Meets Temporal Correspondence: Self-supervised Object-centric Learning in Videoscited by 20
  6. 2023Masked Spatio-Temporal Structure Prediction for Self-supervised Learning on Point Cloud Videoscited by 19
  7. 2023COMEDIAN: Self-Supervised Learning and Knowledge Distillation for Action Spotting using Transformerscited by 17
  8. 2025EndoMamba: An Efficient Foundation Model for Endoscopic Videos via Hierarchical Pre-trainingcited by 11
  9. 2023Betrayed by Attention: A Simple yet Effective Approach for Self-supervised Video Object Segmentationcited by 10
  10. 2024Rethinking Image-to-Video Adaptation: An Object-centric Perspectivecited by 8
  11. 2023Fine-Grained Spatiotemporal Motion Alignment for Contrastive Video Representation Learningcited by 7
  12. 2023DEVIAS: Learning Disentangled Video Representations of Action and Scenecited by 7
11 more
  1. 2023Contrastive Predictive Autoencoders for Dynamic Point Cloud Self-Supervised Learningcited by 5
  2. 2023Bayesian Generalization Error in Linear Neural Networks with Concept Bottleneck Structure and Multitask Formulationcited by 3
  3. 2025PolypSegTrack: Unified Foundation Model for Colonoscopy Video Analysiscited by 3
  4. 2024Upper Bound of Bayesian Generalization Error in Partial Concept Bottleneck Model (CBM): Partial CBM outperforms naive CBMcited by 1
  5. 2025COMIX: Compositional Explanations using Prototypescited by 1
  6. 2024Collaboratively Self-supervised Video Representation Learning for Action Recognitioncited by 0
  7. 2024Self-Supervised Video Representation Learning in a Heuristic Decoupled Perspectivecited by 0
  8. 2025Learning reusable concepts across different egocentric video understanding taskscited by 0
  9. 2025Surgical Foundation Model Leveraging Compression and Entropy Maximization for Image-Guided Surgical Assistancecited by 0
  10. 2026Focus-to-Perceive Representation Learning: A Cognition-Inspired Hierarchical Framework for Endoscopic Video Analysiscited by 0
  11. 2026Impute On-Demand: Adaptive Correlated Time Series Imputation for Changing Environmentscited by 0