mlsanityData to 14 Oct 2026

Memory-Efficient Continual Learning Object Segmentation for Long Video

arXiv 2309.15274 ↗First posted 26 Sep 2023cs.CV cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent state-of-the-art semi-supervised Video Object Segmentation (VOS) methods have shown significant improvements in target object segmentation accuracy when information from preceding frames is used in segmenting the current frame. In particular, such memory-based approaches can help a model to more effectively handle appearance changes (representation drift) or occlusions. Ideally, for maximum performance, Online VOS methods would need all or most of the preceding frames (or their extracted information) to be stored in memory and be used for online learning in later frames. Such a solution is not feasible for long videos, as the required memory size grows without bound, and such methods can fail when memory is limited and a target object experiences repeated representation drifts throughout a video. We propose two novel techniques to reduce the memory requirement of Online VOS methods while improving modeling accuracy and generalization on long videos. Motivated by the success of continual learning techniques in preserving previously-learned knowledge, here we propose Gated-Regularizer Continual Learning (GRCL), which improves the performance of any Online VOS subject to limited memory, and a Reconstruction-based Memory Selection Continual Learning (RMSCL), which empowers Online VOS methods to efficiently benefit from stored information in memory. We also analyze the performance of a hybrid combination of the two proposed methods. Experimental results show that the proposed methods are able to improve the performance of Online VOS models by more than 8%, with improved robustness on long-video datasets while maintaining comparable performance on short-video datasets such as DAVIS16, DAVIS17, and YouTube-VOS18.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 47 of its 80 references that are in the map. First posted on arXiv 26 Sep 2023.

Builds on

47 of 80 references.

References · 47

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  2. 2016Learning without Forgettingcited by 2,496
  3. 2016Progressive Neural Networkscited by 1,361
  4. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
  5. 2019Large Scale Incremental Learningcited by 732
  6. 2017Lifelong Learning with Dynamically Expandable Networkscited by 694
  7. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 681
  8. 2018End-to-End Incremental Learningcited by 597
  9. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 439
  10. 2019Video Object Segmentation using Space-Time Memory Networkscited by 438
  11. 2018YouTube-VOS: A Large-Scale Video Object Segmentation Benchmarkcited by 383
  12. 2021Gradient Projection Memory for Continual Learningcited by 348
35 more
  1. 2019Maintaining Discrimination and Fairness in Class Incremental Learningcited by 283
  2. 2016One-Shot Video Object Segmentationcited by 235
  3. 2018Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselinescited by 228
  4. 2021Associating Objects with Transformers for Video Object Segmentationcited by 226
  5. 2021Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentationcited by 203
  6. 2016Learning Video Object Segmentation from Static Imagescited by 151
  7. 2021Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minimacited by 118
  8. 2020Conditional Channel Gated Networks for Task-Aware Continual Learningcited by 115
  9. 2020Kernelized Memory Network for Video Object Segmentationcited by 111
  10. 2017Online Adaptation of Convolutional Neural Networks for Video Object Segmentationcited by 109
  11. 2020Continual Learning with Node-Importance based Adaptive Group Sparse Regularizationcited by 105
  12. 2020Video Object Segmentation with Adaptive Feature Bank and Uncertain-Region Refinementcited by 101
  13. 2020Overcoming Catastrophic Forgetting in Graph Neural Networkscited by 95
  14. 2020Learning What to Learn for Video Object Segmentationcited by 89
  15. 2019RVOS: End-to-End Recurrent Network for Video Object Segmentationcited by 88
  16. 2021Efficient Regional Memory Network for Video Object Segmentationcited by 86
  17. 2022LVOS: A Benchmark for Long-term Video Object Segmentationcited by 77
  18. 2020Learning Fast and Robust Target Models for Video Object Segmentationcited by 76
  19. 2021Hierarchical Memory Matching Network for Video Object Segmentationcited by 72
  20. 2020Fast Video Object Segmentation using the Global Context Modulecited by 65
  21. 2021Learning Position and Target Consistency for Memory-based Video Object Segmentationcited by 63
  22. 2021Joint Inductive and Transductive Learning for Video Object Segmentationcited by 61
  23. 2018MaskRNN: Instance Level Video Object Segmentationcited by 48
  24. 2018Pseudo-Rehearsal: Achieving Deep Reinforcement Learning without Catastrophic Forgettingcited by 46
  25. 2023Trainable Projected Gradient Method for Robust Fine-tuningcited by 37
  26. 2022Look Before You Match: Instance Understanding Matters in Video Object Segmentationcited by 34
  27. 2021Preventing Catastrophic Forgetting and Distribution Mismatch in Knowledge Distillation via Synthetic Datacited by 32
  28. 2019Continual Learning in Neural Networkscited by 23
  29. 2022Global Spectral Filter Memory Network for Video Object Segmentationcited by 22
  30. 2020Online Continual Learning on Sequencescited by 14
  31. 2019Overcoming Catastrophic Forgetting by Generative Regularizationcited by 13
  32. 2020Addressing Catastrophic Forgetting in Few-Shot Problemscited by 12
  33. 2016Challenges of Feature Selection for Big Data Analyticscited by 8
  34. 2019Meta Learning with Differentiable Closed-form Solver for Fast Video Object Segmentationcited by 6
  35. 2023CLVOS23: A Long Video Object Segmentation Dataset for Continual Learningcited by 3

Built on by

Not cited yet.