mlsanityData to 7 Oct 2026

Memory Storyboard: Leveraging Temporal Segmentation for Streaming Self-Supervised Learning from Egocentric Videos

arXiv 2501.12254 ↗First posted 21 Jan 2025cs.CV cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning holds the promise of learning good representations from real-world continuous uncurated data streams. However, most existing works in visual self-supervised learning focus on static images or artificial data streams. Towards exploring a more realistic learning substrate, we investigate streaming self-supervised learning from long-form real-world egocentric video streams. Inspired by the event segmentation mechanism in human perception and memory, we propose "Memory Storyboard" that groups recent past frames into temporal segments for more effective summarization of the past visual streams for memory replay. To accommodate efficient temporal segmentation, we propose a two-tier memory hierarchy: the recent past is stored in a short-term memory, and the storyboard temporal segments are then transferred to a long-term memory. Experiments on real-world egocentric video datasets including SAYCam and KrishnaCam show that contrastive learning objectives on top of storyboard frames result in semantically meaningful representations that outperform those produced by state-of-the-art unsupervised continual learning methods.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 52 of its 85 references that are in the map. First posted on arXiv 21 Jan 2025.

Builds on

52 of 85 references.

References · 52

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  11. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  12. 2020Supervised Contrastive Learningcited by 2,803
38 more
  1. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  2. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  3. 2018Group Normalizationcited by 1,514
  4. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  5. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  6. 2019Contrastive Multiview Codingcited by 1,297
  7. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  8. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  9. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  10. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  11. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  12. 2019Gradient based sample selection for online continual learningcited by 568
  13. 2017Large Batch Training of Convolutional Networkscited by 529
  14. 2019REMIND Your Neural Network to Prevent Catastrophic Forgettingcited by 197
  15. 2021Self-Supervised Models are Continual Learnerscited by 149
  16. 2019Continual Unsupervised Representation Learningcited by 147
  17. 2016Video Summarization with Long Short-term Memorycited by 142
  18. 2021Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learningcited by 140
  19. 2018Memory Efficient Experience Replay for Streaming Learningcited by 128
  20. 2021Representational Continuity for Unsupervised Continual Learningcited by 95
  21. 2021GCR: Gradient Coreset Based Replay Buffer Selection For Continual Learningcited by 95
  22. 2019Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysiscited by 91
  23. 2018Video Summarization Using Fully Convolutional Sequence Networkscited by 71
  24. 2020Self-supervised learning through the eyes of a childcited by 61
  25. 2023Online Prototype Learning for Online Continual Learningcited by 53
  26. 2023Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled videocited by 40
  27. 2021Continually Learning Self-Supervised Representations with Projected Functional Regularizationcited by 38
  28. 2022The Challenges of Continuous Self-Supervised Learningcited by 36
  29. 2021Wanderlust: Online Continual Object Detection in the Real Worldcited by 34
  30. 2019Unsupervised Progressive Learning and the STAM Architecturecited by 29
  31. 2021How Well Does Self-Supervised Pre-Training Perform with Streaming Data?cited by 29
  32. 2022Label-Efficient Online Continual Object Detection in Streaming Videocited by 16
  33. 2022SCALE: Online Self-Supervised Lifelong Learning without Prior Knowledgecited by 16
  34. 2023Plasticity-Optimized Complementary Networks for Unsupervised Continual Learningcited by 14
  35. 2023Learning from One Continuous Video Streamcited by 11
  36. 2024PooDLe: Pooled and dense self-supervised learning from naturalistic videoscited by 11
  37. 2024Integrating Present and Past in Unsupervised Continual Learningcited by 8
  38. 2023Revisiting Kernel Temporal Segmentation as an Adaptive Tokenizer for Long-form Video Understandingcited by 7

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Temporal Straightening for Latent Planningcited by 32
  2. 2026From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanismscited by 19
  3. 2026EgoEverything: A Benchmark for Human Behavior Inspired Long Context Egocentric Video Understanding in AR Environmentcited by 0
  4. 2026Continual Visual and Verbal Learning Through a Child's Egocentric Inputcited by 0
  5. 2026Lifelong Representations: A Survey on Continual Self-Supervised Learning for Vision Modelscited by 0
  6. 2026I Have a Stream: Making Self-Supervised Learning Work on Continuous Videocited by 0