# Selfless Sequential Learning

arXiv: https://arxiv.org/abs/1806.05421 · HTML: https://mlsanity.com/paper/1806.05421/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jun 2018.

## Built on by (most cited first, 30 of 44)

- [Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations](https://mlsanity.com/paper/2103.06342/) (2021)
- [Learning to Unlearn: Instance-wise Unlearning for Pre-trained Classifiers](https://mlsanity.com/paper/2301.11578/) (2023)
- [Advances and Challenges in Meta-Learning: A Technical Review](https://mlsanity.com/paper/2307.04722/) (2023)
- [PIVOT: Prompting for Video Continual Learning](https://mlsanity.com/paper/2212.04842/) (2022)
- [Memory Efficient Continual Learning with Transformers](https://mlsanity.com/paper/2203.04640/) (2022)
- [EgoObjects: A Large-Scale Egocentric Dataset for Fine-Grained Object Understanding](https://mlsanity.com/paper/2309.08816/) (2023)
- [Understanding Continual Learning Settings with Data Distribution Drift Analysis](https://mlsanity.com/paper/2104.01678/) (2021)
- [Continual Learning for Recurrent Neural Networks: an Empirical Evaluation](https://mlsanity.com/paper/2103.07492/) (2021)
- [Learning Bayesian Sparse Networks with Full Experience Replay for Continual Learning](https://mlsanity.com/paper/2202.10203/) (2022)
- [Continual Learning on Noisy Data Streams via Self-Purified Replay](https://mlsanity.com/paper/2110.07735/) (2021)
- [An Investigation of Replay-based Approaches for Continual Learning](https://mlsanity.com/paper/2108.06758/) (2021)
- [Optimizing Reusable Knowledge for Continual Learning via Metalearning](https://mlsanity.com/paper/2106.05390/) (2021)
- [Continual Object Detection via Prototypical Task Correlation Guided Gating Mechanism](https://mlsanity.com/paper/2205.03055/) (2022)
- [Sparsity and Heterogeneous Dropout for Continual Learning in the Null Space of Neural Activations](https://mlsanity.com/paper/2203.06514/) (2022)
- [Continual Learning with Transformers for Image Classification](https://mlsanity.com/paper/2206.14085/) (2022)
- [Optimizers Qualitatively Alter Solutions And We Should Leverage This](https://mlsanity.com/paper/2507.12224/) (2025)
- [SPeCiaL: Self-Supervised Pretraining for Continual Learning](https://mlsanity.com/paper/2106.09065/) (2021)
- [Continual Learning for Predictive Maintenance: Overview and Challenges](https://mlsanity.com/paper/2301.12467/) (2023)
- [Elephant Neural Networks: Born to Be a Continual Learner](https://mlsanity.com/paper/2310.01365/) (2023)
- [Memory-efficient Reinforcement Learning with Value-based Knowledge Consolidation](https://mlsanity.com/paper/2205.10868/) (2022)
- [Continual Competitive Memory: A Neural System for Online Task-Free Lifelong Learning](https://mlsanity.com/paper/2106.13300/) (2021)
- [CNLL: A Semi-supervised Approach For Continual Noisy Label Learning](https://mlsanity.com/paper/2204.09881/) (2022)
- [PECTP: Parameter-Efficient Cross-Task Prompts for Incremental Vision Transformer](https://mlsanity.com/paper/2407.03813/) (2024)
- [Continual Reinforcement Learning by Planning with Online World Models](https://mlsanity.com/paper/2507.09177/) (2025)
- [Overcoming Catastrophic Forgetting with Gaussian Mixture Replay](https://mlsanity.com/paper/2104.09220/) (2021)
- [Continual Learning with Fully Probabilistic Models](https://mlsanity.com/paper/2104.09240/) (2021)
- [Adversarial Counterfactual Augmentation: Application in Alzheimer's Disease Classification](https://mlsanity.com/paper/2203.07815/) (2022)
- [Continual Learning of Dynamical Systems with Competitive Federated Reservoir Computing](https://mlsanity.com/paper/2206.13336/) (2022)
- [Continual Learning for Tumor Classification in Histopathology Images](https://mlsanity.com/paper/2208.03609/) (2022)
- [CLR: Channel-wise Lightweight Reprogramming for Continual Learning](https://mlsanity.com/paper/2307.11386/) (2023)

## Abstract (arXiv)

Sequential learning, also called lifelong learning, studies the problem of learning tasks in a sequence with access restricted to only the data of the current task. In this paper we look at a scenario with fixed model capacity, and postulate that the learning process should not be selfish, i.e. it should account for future tasks to be added and thus leave enough capacity for them. To achieve Selfless Sequential Learning we study different regularization strategies and activation functions. We find that imposing sparsity at the level of the representation (i.e.~neuron activations) is more beneficial for sequential learning than encouraging parameter sparsity. In particular, we propose a novel regularizer, that encourages representation sparsity by means of neural inhibition. It results in few active neurons which in turn leaves more free neurons to be utilized by upcoming tasks. As neural inhibition over an entire layer can be too drastic, especially for complex tasks requiring strong representations, our regularizer only inhibits other neurons in a local neighbourhood, inspired by lateral inhibition processes in the brain. We combine our novel regularizer, with state-of-the-art lifelong learning methods that penalize changes to important previously learned parts of the network. We show that our new regularizer leads to increased sparsity which translates in consistent performance improvement %over alternative regularizers we studied on diverse datasets.