mlsanityData to 7 Oct 2026

Memory-augmented Dense Predictive Coding for Video Representation Learning

arXiv 2008.01065 ↗First posted 3 Aug 2020cs.CVcited by 137

Not on the map. Why?

Abstract (arXiv)

The objective of this paper is self-supervised learning from video, in particular for representations for action recognition. We make the following contributions: (i) We propose a new architecture and learning framework Memory-augmented Dense Predictive Coding (MemDPC) for the task. It is trained with a predictive attention mechanism over the set of compressed memories, such that any future states can always be constructed by a convex combination of the condense representations, allowing to make multiple hypotheses efficiently. (ii) We investigate visual-only self-supervised video representation learning from RGB frames, or from unsupervised optical flow, or both. (iii) We thoroughly evaluate the quality of learnt representation on four different downstream tasks: action recognition, video retrieval, learning with scarce annotations, and unintentional action classification. In all cases, we demonstrate state-of-the-art or comparable performance over other approaches with orders of magnitude fewer training data.

Key facts. Cited by 137 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Aug 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 54; 2022: 39; 2023: 19; 2024: 17; 2025: 6; 2026: 2.

137 papers.

Sort:

By area:

  1. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  2. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  3. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  4. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  5. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  6. 2021Anticipative Video Transformercited by 197
  7. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  8. 2024Enhancing End-to-End Autonomous Driving with Latent World Modelcited by 161
  9. 2021TCLR: Temporal Contrastive Learning for Video Representationcited by 136
  10. 2021Broaden Your Views for Self-Supervised Video Learningcited by 108
  11. 2022Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learningcited by 90
  12. 2021Divide and Contrast: Self-supervised Learning from Uncurated Datacited by 85
38 more
  1. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  2. 2021Robust Audio-Visual Instance Discriminationcited by 80
  3. 2022Bridging Video-text Retrieval with Multiple Choice Questionscited by 79
  4. 2021Skeleton-Contrastive 3D Action Representation Learningcited by 76
  5. 2022TCTrack: Temporal Contexts for Aerial Trackingcited by 69
  6. 2021Video Prediction Recalling Long-term Motion Context via Memory Alignment Learningcited by 62
  7. 2021Semi-Supervised Action Recognition with Temporal Contrastive Learningcited by 50
  8. 2021Towards Unified Surgical Skill Assessmentcited by 49
  9. 2021Learning the Predictability of the Futurecited by 48
  10. 2021Motion-aware Contrastive Video Representation Learning via Foreground-background Mergingcited by 45
  11. 2024Memory Consolidation Enables Long-Context Video Understandingcited by 45
  12. 2021Time-Equivariant Contrastive Video Representation Learningcited by 44
  13. 2021Learning to Anticipate Egocentric Actions by Imaginationcited by 40
  14. 2021Video Contrastive Learning with Global Contextcited by 40
  15. 2021Shot Contrastive Self-Supervised Learning for Scene Boundary Detectioncited by 35
  16. 2021ASCNet: Self-supervised Video Representation Learning with Appearance-Speed Consistencycited by 35
  17. 2021Self-supervised Video Representation Learning by Context and Motion Decouplingcited by 34
  18. 2023Diverse Human Motion Prediction Guided by Multi-Level Spatial-Temporal Anchorscited by 34
  19. 2021Space-Time Crop & Attend: Improving Cross-modal Video Representation Learningcited by 31
  20. 2021TCGL: Temporal Contrastive Graph for Self-supervised Video Representation Learningcited by 31
  21. 2022Unsupervised Pre-training for Temporal Action Localization Taskscited by 31
  22. 2024Scaling 4D Representationscited by 30
  23. 2021Self-supervised object detection from audio-visual correspondencecited by 29
  24. 2022MILES: Visual BERT Pre-training with Injected Language Semantics for Video-text Retrievalcited by 29
  25. 2022MAR: Masked Autoencoders for Efficient Action Recognitioncited by 29
  26. 2021Self-Supervised Video Representation Learning with Meta-Contrastive Networkcited by 28
  27. 2024Universal Time-Series Representation Learning: A Surveycited by 28
  28. 2021Enhancing Self-supervised Video Representation Learning via Multi-level Feature Optimizationcited by 27
  29. 2022SMEMO: Social Memory for Trajectory Forecastingcited by 26
  30. 2022Locality-Aware Inter-and Intra-Video Reconstruction for Self-Supervised Correspondence Learningcited by 26
  31. 2023Towards Real-World Visual Tracking with Temporal Contextscited by 26
  32. 2023MEMTO: Memory-guided Transformer for Multivariate Time Series Anomaly Detectioncited by 26
  33. 2024VideoMamba: Spatio-Temporal Selective State Space Modelcited by 25
  34. 2021Self-supervised Representation Learning Framework for Remote Physiological Measurement Using Spatiotemporal Augmentation Losscited by 23
  35. 2022Motion-Focused Contrastive Learning of Video Representationscited by 23
  36. 2022Static and Dynamic Concepts for Self-supervised Video Representation Learningcited by 23
  37. 2022Learning State-Aware Visual Representations from Audible Interactionscited by 23
  38. 2021Learning Representational Invariances for Data-Efficient Action Recognitioncited by 22

50 most cited of 137 All 137 (JSON)