# Rethinking Task-Incremental Learning Baselines

arXiv: https://arxiv.org/abs/2205.11367 · HTML: https://mlsanity.com/paper/2205.11367/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 43 references that are in the map.
- First posted on arXiv 23 May 2022.

## Builds on (most cited first, 30 of 30)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [PointCNN: Convolution On X-Transformed Points](https://mlsanity.com/paper/1801.07791/) (2018)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference](https://mlsanity.com/paper/1810.11910/) (2018)
- [Expert Gate: Lifelong Learning with a Network of Experts](https://mlsanity.com/paper/1611.06194/) (2016)
- [PathNet: Evolution Channels Gradient Descent in Super Neural Networks](https://mlsanity.com/paper/1701.08734/) (2017)
- [Memory Networks](https://mlsanity.com/paper/1410.3916/) (2014)
- [Variational Continual Learning](https://mlsanity.com/paper/1710.10628/) (2017)
- [Mnemonics Training: Multi-Class Incremental Learning without Forgetting](https://mlsanity.com/paper/2002.10211/) (2020)
- [FearNet: Brain-Inspired Model for Incremental Learning](https://mlsanity.com/paper/1711.10563/) (2017)
- [Neural Episodic Control](https://mlsanity.com/paper/1703.01988/) (2017)
- [Adversarial Continual Learning](https://mlsanity.com/paper/2003.09553/) (2020)
- [Overcoming Catastrophic Forgetting with Unlabeled Data in the Wild](https://mlsanity.com/paper/1903.12648/) (2019)
- [Uncertainty-guided Continual Learning with Bayesian Neural Networks](https://mlsanity.com/paper/1906.02425/) (2019)
- [Rethinking Experience Replay: a Bag of Tricks for Continual Learning](https://mlsanity.com/paper/2010.05595/) (2020)
- [One-shot Learning with Memory-Augmented Neural Networks](https://mlsanity.com/paper/1605.06065/) (2016)
- [Growing a Brain: Fine-Tuning by Increasing Model Capacity](https://mlsanity.com/paper/1907.07844/) (2019)
- [Graph-Based Continual Learning](https://mlsanity.com/paper/2007.04813/) (2020)
- [Learning without Forgetting for 3D Point Cloud Objects](https://mlsanity.com/paper/2106.14275/) (2021)

## Built on by (most cited first, 4 of 4)

- [Online Continual Learning For Interactive Instruction Following Agents](https://mlsanity.com/paper/2403.07548/) (2024)
- [ViLCo-Bench: VIdeo Language COntinual learning Benchmark](https://mlsanity.com/paper/2406.13123/) (2024)
- [H2ST: Hierarchical Two-Sample Tests for Continual Out-of-Distribution Detection](https://mlsanity.com/paper/2503.14832/) (2025)
- [FETCH: A Memory-Efficient Replay Approach for Continual Learning in Image Classification](https://mlsanity.com/paper/2407.12375/) (2024)

## Abstract (arXiv)

It is common to have continuous streams of new data that need to be introduced in the system in real-world applications. The model needs to learn newly added capabilities (future tasks) while retaining the old knowledge (past tasks). Incremental learning has recently become increasingly appealing for this problem. Task-incremental learning is a kind of incremental learning where task identity of newly included task (a set of classes) remains known during inference. A common goal of task-incremental methods is to design a network that can operate on minimal size, maintaining decent performance. To manage the stability-plasticity dilemma, different methods utilize replay memory of past tasks, specialized hardware, regularization monitoring etc. However, these methods are still less memory efficient in terms of architecture growth or input data costs. In this study, we present a simple yet effective adjustment network (SAN) for task incremental learning that achieves near state-of-the-art performance while using minimal architectural size without using memory instances compared to previous state-of-the-art approaches. We investigate this approach on both 3D point cloud object (ModelNet40) and 2D image (CIFAR10, CIFAR100, MiniImageNet, MNIST, PermutedMNIST, notMNIST, SVHN, and FashionMNIST) recognition tasks and establish a strong baseline result for a fair comparison with existing methods. On both 2D and 3D domains, we also observe that SAN is primarily unaffected by different task orders in a task-incremental setting.