# A baseline on continual learning methods for video action recognition

arXiv: https://arxiv.org/abs/2304.10335 · HTML: https://mlsanity.com/paper/2304.10335/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 28 references that are in the map.
- First posted on arXiv 20 Apr 2023.

## Builds on (most cited first, 20 of 20)

- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [Gradient based sample selection for online continual learning](https://mlsanity.com/paper/1903.08671/) (2019)
- [Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference](https://mlsanity.com/paper/1810.11910/) (2018)
- [Progress & Compress: A scalable framework for continual learning](https://mlsanity.com/paper/1805.06370/) (2018)
- [Using Hindsight to Anchor Past Knowledge in Continual Learning](https://mlsanity.com/paper/2002.08165/) (2020)
- [Continual Learning via Neural Pruning](https://mlsanity.com/paper/1903.04476/) (2019)
- [Measuring and regularizing networks in function space](https://mlsanity.com/paper/1805.08289/) (2018)
- [Class-Incremental Learning for Action Recognition in Videos](https://mlsanity.com/paper/2203.13611/) (2022)
- [Transfer without Forgetting](https://mlsanity.com/paper/2206.00388/) (2022)
- [TCGL: Temporal Contrastive Graph for Self-supervised Video Representation Learning](https://mlsanity.com/paper/2112.03587/) (2021)

## Built on by (most cited first, 1 of 1)

- [Bisecle: Binding and Separation in Continual Learning for Video Language Understanding](https://mlsanity.com/paper/2507.00469/) (2025)

## Abstract (arXiv)

Continual learning has recently attracted attention from the research community, as it aims to solve long-standing limitations of classic supervisedly-trained models. However, most research on this subject has tackled continual learning in simple image classification scenarios. In this paper, we present a benchmark of state-of-the-art continual learning methods on video action recognition. Besides the increased complexity due to the temporal dimension, the video setting imposes stronger requirements on computing resources for top-performing rehearsal methods. To counteract the increased memory requirements, we present two method-agnostic variants for rehearsal methods, exploiting measures of either model confidence or data information to select memorable samples. Our experiments show that, as expected from the literature, rehearsal methods outperform other approaches; moreover, the proposed memory-efficient variants are shown to be effective at retaining a certain level of performance with a smaller buffer size.