# When Video Classification Meets Incremental Classes

arXiv: https://arxiv.org/abs/2106.15827 · HTML: https://mlsanity.com/paper/2106.15827/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 66 references that are in the map.
- First posted on arXiv 30 Jun 2021.

## Builds on (most cited first, 30 of 43)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [Convolutional Two-Stream Network Fusion for Video Action Recognition](https://mlsanity.com/paper/1604.06573/) (2016)
- [Temporal Relational Reasoning in Videos](https://mlsanity.com/paper/1711.08496/) (2017)
- [Video Classification with Channel-Separated Convolutional Networks](https://mlsanity.com/paper/1904.02811/) (2019)
- [Semantic Drift Compensation for Class-Incremental Learning](https://mlsanity.com/paper/2004.00440/) (2020)
- [Mnemonics Training: Multi-Class Incremental Learning without Forgetting](https://mlsanity.com/paper/2002.10211/) (2020)
- [TEA: Temporal Excitation and Aggregation for Action Recognition](https://mlsanity.com/paper/2004.01398/) (2020)
- [STM: SpatioTemporal and Motion Encoding for Action Recognition](https://mlsanity.com/paper/1908.02486/) (2019)
- [Few-Shot Video Classification via Temporal Alignment](https://mlsanity.com/paper/1906.11415/) (2019)
- [ECO: Efficient Convolutional Network for Online Video Understanding](https://mlsanity.com/paper/1804.09066/) (2018)
- [Memory-Efficient Incremental Learning Through Feature Adaptation](https://mlsanity.com/paper/2004.00713/) (2020)
- [TEINet: Towards an Efficient Architecture for Video Recognition](https://mlsanity.com/paper/1911.09435/) (2019)

## Built on by (most cited first, 18 of 18)

- [Class-Incremental Learning: A Survey](https://mlsanity.com/paper/2302.03648/) (2023)
- [Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Need](https://mlsanity.com/paper/2303.07338/) (2023)
- [A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning](https://mlsanity.com/paper/2205.13218/) (2022)
- [Recent Advances of Continual Learning in Computer Vision: An Overview](https://mlsanity.com/paper/2109.11369/) (2021)
- [vCLIMB: A Novel Video Class Incremental Learning Benchmark](https://mlsanity.com/paper/2201.09381/) (2022)
- [RBC: Rectifying the Biased Context in Continual Semantic Segmentation](https://mlsanity.com/paper/2203.08404/) (2022)
- [Online Continual Learning For Interactive Instruction Following Agents](https://mlsanity.com/paper/2403.07548/) (2024)
- [Generative Negative Text Replay for Continual Vision-Language Pretraining](https://mlsanity.com/paper/2210.17322/) (2022)
- [Learning a Condensed Frame for Memory-Efficient Video Class-Incremental Learning](https://mlsanity.com/paper/2211.00833/) (2022)
- [Just a Glimpse: Rethinking Temporal Information for Video Continual Learning](https://mlsanity.com/paper/2305.18418/) (2023)
- [MGH: Metadata Guided Hypergraph Modeling for Unsupervised Person Re-identification](https://mlsanity.com/paper/2110.05886/) (2021)
- [LW2G: Learning Whether to Grow for Prompt-based Continual Learning](https://mlsanity.com/paper/2409.18860/) (2024)
- [PECTP: Parameter-Efficient Cross-Task Prompts for Incremental Vision Transformer](https://mlsanity.com/paper/2407.03813/) (2024)
- [Deep Class Incremental Learning from Decentralized Data](https://mlsanity.com/paper/2203.05984/) (2022)
- [Teacher Agent: A Knowledge Distillation-Free Framework for Rehearsal-based Video Incremental Learning](https://mlsanity.com/paper/2306.00393/) (2023)
- [Progressive Classifier and Feature Extractor Adaptation for Unsupervised Domain Adaptation on Point Clouds](https://mlsanity.com/paper/2311.16474/) (2023)
- [CRISP: Contrastive Residual Injection and Semantic Prompting for Continual Video Instance Segmentation](https://mlsanity.com/paper/2508.10432/) (2025)
- [ESSENTIAL: Episodic and Semantic Memory Integration for Video Class-Incremental Learning](https://mlsanity.com/paper/2508.10896/) (2025)

## Abstract (arXiv)

With the rapid development of social media, tremendous videos with new classes are generated daily, which raise an urgent demand for video classification methods that can continuously update new classes while maintaining the knowledge of old videos with limited storage and computing resources. In this paper, we summarize this task as Class-Incremental Video Classification (CIVC) and propose a novel framework to address it. As a subarea of incremental learning tasks, the challenge of catastrophic forgetting is unavoidable in CIVC. To better alleviate it, we utilize some characteristics of videos. First, we decompose the spatio-temporal knowledge before distillation rather than treating it as a whole in the knowledge transfer process; trajectory is also used to refine the decomposition. Second, we propose a dual granularity exemplar selection method to select and store representative video instances of old classes and key-frames inside videos under a tight storage budget. We benchmark our method and previous SOTA class-incremental learning methods on Something-Something V2 and Kinetics datasets, and our method outperforms previous methods significantly.