# vCLIMB: A Novel Video Class Incremental Learning Benchmark

arXiv: https://arxiv.org/abs/2201.09381 · HTML: https://mlsanity.com/paper/2201.09381/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 45 references that are in the map.
- First posted on arXiv 23 Jan 2022.

## Builds on (most cited first, 30 of 32)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [Fully-Convolutional Siamese Networks for Object Tracking](https://mlsanity.com/paper/1606.09549/) (2016)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning](https://mlsanity.com/paper/1606.04586/) (2016)
- [DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition](https://mlsanity.com/paper/1310.1531/) (2013)
- [Differentiable Augmentation for Data-Efficient GAN Training](https://mlsanity.com/paper/2006.10738/) (2020)
- [Temporal Segment Networks for Action Recognition in Videos](https://mlsanity.com/paper/1705.02953/) (2017)
- [Learning without Memorizing](https://mlsanity.com/paper/1811.08051/) (2018)
- [Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines](https://mlsanity.com/paper/1810.12488/) (2018)
- [Mnemonics Training: Multi-Class Incremental Learning without Forgetting](https://mlsanity.com/paper/2002.10211/) (2020)
- [Modeling the Background for Incremental Learning in Semantic Segmentation](https://mlsanity.com/paper/2002.00718/) (2020)
- [Learning to Remember: A Synaptic Plasticity Driven Framework for Continual Learning](https://mlsanity.com/paper/1904.03137/) (2019)
- [FearNet: Brain-Inspired Model for Incremental Learning](https://mlsanity.com/paper/1711.10563/) (2017)
- [Consistency Regularization for Generative Adversarial Networks](https://mlsanity.com/paper/1910.12027/) (2019)
- [Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognition](https://mlsanity.com/paper/1912.05534/) (2019)

## Built on by (most cited first, 28 of 28)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Class-Incremental Learning: A Survey](https://mlsanity.com/paper/2302.03648/) (2023)
- [Real-Time Evaluation in Online Continual Learning: A New Hope](https://mlsanity.com/paper/2302.01047/) (2023)
- [PIVOT: Prompting for Video Continual Learning](https://mlsanity.com/paper/2212.04842/) (2022)
- [Online Continual Learning For Interactive Instruction Following Agents](https://mlsanity.com/paper/2403.07548/) (2024)
- [Continual Text-to-Video Retrieval with Frame Fusion and Task-Aware Routing](https://mlsanity.com/paper/2503.10111/) (2025)
- [Learning a Condensed Frame for Memory-Efficient Video Class-Incremental Learning](https://mlsanity.com/paper/2211.00833/) (2022)
- [Just a Glimpse: Rethinking Temporal Information for Video Continual Learning](https://mlsanity.com/paper/2305.18418/) (2023)
- [Balanced Destruction-Reconstruction Dynamics for Memory-replay Class Incremental Learning](https://mlsanity.com/paper/2308.01698/) (2023)
- [RainbowPrompt: Diversity-Enhanced Prompt-Evolving for Continual Learning](https://mlsanity.com/paper/2507.22553/) (2025)
- [ViLCo-Bench: VIdeo Language COntinual learning Benchmark](https://mlsanity.com/paper/2406.13123/) (2024)
- [StPR: Spatiotemporal Preservation and Routing for Exemplar-Free Video Class-Incremental Learning](https://mlsanity.com/paper/2505.13997/) (2025)
- [Coherent Temporal Synthesis for Incremental Action Segmentation](https://mlsanity.com/paper/2403.06102/) (2024)
- [CSTA: Spatial-Temporal Causal Adaptive Learning for Exemplar-Free Video Class-Incremental Learning](https://mlsanity.com/paper/2501.07236/) (2025)
- [Teacher Agent: A Knowledge Distillation-Free Framework for Rehearsal-based Video Incremental Learning](https://mlsanity.com/paper/2306.00393/) (2023)
- [Hierarchical Augmentation and Distillation for Class Incremental Audio-Visual Video Recognition](https://mlsanity.com/paper/2401.06287/) (2024)
- [Lifelong Learning of Video Diffusion Models From a Single Video Stream](https://mlsanity.com/paper/2406.04814/) (2024)
- [CRISP: Contrastive Residual Injection and Semantic Prompting for Continual Video Instance Segmentation](https://mlsanity.com/paper/2508.10432/) (2025)
- [Decoupled Prompt-Adapter Tuning for Continual Activity Recognition](https://mlsanity.com/paper/2407.14811/) (2024)
- [CRAM: Large-scale Video Continual Learning with Bootstrapped Compression](https://mlsanity.com/paper/2508.05001/) (2025)
- [ESSENTIAL: Episodic and Semantic Memory Integration for Video Class-Incremental Learning](https://mlsanity.com/paper/2508.10896/) (2025)
- [Forget Less, Retain More: A Lightweight Regularizer for Rehearsal-Based Continual Learning](https://mlsanity.com/paper/2512.01818/) (2025)
- [Slightly Shift New Classes to Remember Old Classes for Video Class-Incremental Learning](https://mlsanity.com/paper/2404.00901/) (2024)
- [Continual Learning for Multiple Modalities](https://mlsanity.com/paper/2503.08064/) (2025)
- [Unsupervised Video Continual Learning via Non-Parametric Deep Embedded Clustering](https://mlsanity.com/paper/2508.21773/) (2025)
- [Unsupervised Video Class-Incremental Learning via Deep Embedded Clustering Management](https://mlsanity.com/paper/2601.14069/) (2026)
- [Preserving Knowledge across Space and Time for Continual Video Deepfake Detection](https://mlsanity.com/paper/2609.03446/) (2026)
- [CE4L: Continual Ego, Exo, and Ego-Exo Learning](https://mlsanity.com/paper/2609.23492/) (2026)

## Abstract (arXiv)

Continual learning (CL) is under-explored in the video domain. The few existing works contain splits with imbalanced class distributions over the tasks, or study the problem in unsuitable datasets. We introduce vCLIMB, a novel video continual learning benchmark. vCLIMB is a standardized test-bed to analyze catastrophic forgetting of deep models in video continual learning. In contrast to previous work, we focus on class incremental continual learning with models trained on a sequence of disjoint tasks, and distribute the number of classes uniformly across the tasks. We perform in-depth evaluations of existing CL methods in vCLIMB, and observe two unique challenges in video data. The selection of instances to store in episodic memory is performed at the frame level. Second, untrimmed training data influences the effectiveness of frame sampling strategies. We address these two challenges by proposing a temporal consistency regularization that can be applied on top of memory-based continual learning methods. Our approach significantly improves the baseline, by up to 24% on the untrimmed continual learning task.