# Is Multi-Task Learning an Upper Bound for Continual Learning?

arXiv: https://arxiv.org/abs/2210.14797 · HTML: https://mlsanity.com/paper/2210.14797/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 22 references that are in the map.
- First posted on arXiv 26 Oct 2022.

## Builds on (most cited first, 17 of 17)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Three scenarios for continual learning](https://mlsanity.com/paper/1904.07734/) (2019)
- [Progress & Compress: A scalable framework for continual learning](https://mlsanity.com/paper/1805.06370/) (2018)
- [Orthogonal Gradient Descent for Continual Learning](https://mlsanity.com/paper/1910.07104/) (2019)
- [Co2L: Contrastive Continual Learning](https://mlsanity.com/paper/2106.14413/) (2021)
- [Supermasks in Superposition](https://mlsanity.com/paper/2006.14769/) (2020)
- [Self-Supervised Models are Continual Learners](https://mlsanity.com/paper/2112.04215/) (2021)
- [Linear Mode Connectivity in Multitask and Continual Learning](https://mlsanity.com/paper/2010.04495/) (2020)
- [Continually Learning Self-Supervised Representations with Projected Functional Regularization](https://mlsanity.com/paper/2112.15022/) (2021)
- [Generative Continual Concept Learning](https://mlsanity.com/paper/1906.03744/) (2019)
- [Lifelong Learning with Sketched Structural Regularization](https://mlsanity.com/paper/2104.08604/) (2021)

## Built on by (most cited first, 6 of 6)

- [The Ideal Continual Learner: An Agent That Never Forgets](https://mlsanity.com/paper/2305.00316/) (2023)
- [Towards a Unified Framework for Adaptable Problematic Content Detection via Continual Learning](https://mlsanity.com/paper/2309.16905/) (2023)
- [T2I-ConBench: Text-to-Image Benchmark for Continual Post-training](https://mlsanity.com/paper/2505.16875/) (2025)
- [Learning with Preserving for Continual Multitask Learning](https://mlsanity.com/paper/2511.11676/) (2025)
- [Retrofit: Continual Learning with Controlled Forgetting for Binary Security Detection and Analysis](https://mlsanity.com/paper/2511.11439/) (2025)
- [Decoupling Knowledge and Privacy: Post-Task Self-Distillation Replay for LLM Continual Learning](https://mlsanity.com/paper/2609.29711/) (2026)

## Abstract (arXiv)

Continual and multi-task learning are common machine learning approaches to learning from multiple tasks. The existing works in the literature often assume multi-task learning as a sensible performance upper bound for various continual learning algorithms. While this assumption is empirically verified for different continual learning benchmarks, it is not rigorously justified. Moreover, it is imaginable that when learning from multiple tasks, a small subset of these tasks could behave as adversarial tasks reducing the overall learning performance in a multi-task setting. In contrast, continual learning approaches can avoid the performance drop caused by such adversarial tasks to preserve their performance on the rest of the tasks, leading to better performance than a multi-task learner. This paper proposes a novel continual self-supervised learning setting, where each task corresponds to learning an invariant representation for a specific class of data augmentations. In this setting, we show that continual learning often beats multi-task learning on various benchmark datasets, including MNIST, CIFAR-10, and CIFAR-100.