# Learnability and Algorithm for Continual Learning

arXiv: https://arxiv.org/abs/2306.12646 · HTML: https://mlsanity.com/paper/2306.12646/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 65 of its 77 references that are in the map.
- First posted on arXiv 22 Jun 2023.

## Builds on (most cited first, 30 of 65)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [Three scenarios for continual learning](https://mlsanity.com/paper/1904.07734/) (2019)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Progress & Compress: A scalable framework for continual learning](https://mlsanity.com/paper/1805.06370/) (2018)
- [ViM: Out-Of-Distribution with Virtual-logit Matching](https://mlsanity.com/paper/2203.10807/) (2022)
- [Online Continual Learning with Maximally Interfered Retrieval](https://mlsanity.com/paper/1908.04742/) (2019)
- [Expert Gate: Lifelong Learning with a Network of Experts](https://mlsanity.com/paper/1611.06194/) (2016)
- [DER: Dynamically Expandable Representation for Class Incremental Learning](https://mlsanity.com/paper/2103.16788/) (2021)
- [On Tiny Episodic Memories in Continual Learning](https://mlsanity.com/paper/1902.10486/) (2019)
- [Continual learning with hypernetworks](https://mlsanity.com/paper/1906.00695/) (2019)
- [Learning without Memorizing](https://mlsanity.com/paper/1811.08051/) (2018)
- [Co2L: Contrastive Continual Learning](https://mlsanity.com/paper/2106.14413/) (2021)
- [Supermasks in Superposition](https://mlsanity.com/paper/2006.14769/) (2020)
- [Mnemonics Training: Multi-Class Incremental Learning without Forgetting](https://mlsanity.com/paper/2002.10211/) (2020)

## Built on by (most cited first, 28 of 28)

- [A Unified Approach to Domain Incremental Learning with Memory: Theory and Algorithm](https://mlsanity.com/paper/2310.12244/) (2023)
- [HiDe-LLaVA: Hierarchical Decoupling for Continual Instruction Tuning of Multimodal Large Language Model](https://mlsanity.com/paper/2503.12941/) (2025)
- [Class Incremental Learning via Likelihood Ratio Based Task Prediction](https://mlsanity.com/paper/2309.15048/) (2023)
- [PILoRA: Prototype Guided Incremental LoRA for Federated Class-Incremental Learning](https://mlsanity.com/paper/2401.02094/) (2024)
- [Class Incremental Learning with Self-Supervised Pre-Training and Prototype Learning](https://mlsanity.com/paper/2308.02346/) (2023)
- [Out-of-Distribution Generalization in Time Series: A Survey](https://mlsanity.com/paper/2503.13868/) (2025)
- [Addressing Imbalanced Domain-Incremental Learning through Dual-Balance Collaborative Experts](https://mlsanity.com/paper/2507.07100/) (2025)
- [MCITlib: Multimodal Continual Instruction Tuning Library and Benchmark](https://mlsanity.com/paper/2508.07307/) (2025)
- [Replay-and-Forget-Free Graph Class-Incremental Learning: A Task Profiling and Prompting Approach](https://mlsanity.com/paper/2410.10341/) (2024)
- [Federated Continual Instruction Tuning](https://mlsanity.com/paper/2503.12897/) (2025)
- [Continual Evidential Deep Learning for Out-of-Distribution Detection](https://mlsanity.com/paper/2309.02995/) (2023)
- [PASS++: A Dual Bias Reduction Framework for Non-Exemplar Class-Incremental Learning](https://mlsanity.com/paper/2407.14029/) (2024)
- [On the Discrimination and Consistency for Exemplar-Free Class Incremental Learning](https://mlsanity.com/paper/2501.15454/) (2025)
- [Sub-network Discovery and Soft-masking for Continual Learning of Mixed Tasks](https://mlsanity.com/paper/2310.09436/) (2023)
- [In-context Continual Learning Assisted by an External Continual Learner](https://mlsanity.com/paper/2412.15563/) (2024)
- [AnaCP: Toward Upper-Bound Continual Learning via Analytic Contrastive Projection](https://mlsanity.com/paper/2511.13880/) (2025)
- [Continual Learning Using Only Large Language Model Prompting](https://mlsanity.com/paper/2412.15479/) (2024)
- [Active Learning for Continual Learning: Keeping the Past Alive in the Present](https://mlsanity.com/paper/2501.14278/) (2025)
- [Continual Hyperbolic Learning of Instances and Classes](https://mlsanity.com/paper/2506.10710/) (2025)
- [DESIRE: Dynamic Knowledge Consolidation for Rehearsal-Free Continual Learning](https://mlsanity.com/paper/2411.19154/) (2024)
- [Achieving Upper Bound Accuracy of Joint Training in Continual Learning](https://mlsanity.com/paper/2502.12388/) (2025)
- [Global Convergence of Continual Learning on Non-IID Data](https://mlsanity.com/paper/2503.18511/) (2025)
- [Humans Coexist, So Must Embodied Artificial Agents](https://mlsanity.com/paper/2502.04809/) (2025)
- [Buffer-free Class-Incremental Learning with Out-of-Distribution Detection](https://mlsanity.com/paper/2505.23412/) (2025)
- [CL-VISTA: Benchmarking Continual Learning in Video Large Language Models](https://mlsanity.com/paper/2604.00677/) (2026)
- [Balancing the Causal Effects in Class-Incremental Learning](https://mlsanity.com/paper/2402.10063/) (2024)
- [On Understanding of the Dynamics of Model Capacity in Continual Learning](https://mlsanity.com/paper/2508.08052/) (2025)
- [Exploring the Impact of Parameter Update Magnitude on Forgetting and Generalization of Continual Learning](https://mlsanity.com/paper/2602.20796/) (2026)

## Abstract (arXiv)

This paper studies the challenging continual learning (CL) setting of Class Incremental Learning (CIL). CIL learns a sequence of tasks consisting of disjoint sets of concepts or classes. At any time, a single model is built that can be applied to predict/classify test instances of any classes learned thus far without providing any task related information for each test instance. Although many techniques have been proposed for CIL, they are mostly empirical. It has been shown recently that a strong CIL system needs a strong within-task prediction (WP) and a strong out-of-distribution (OOD) detection for each task. However, it is still not known whether CIL is actually learnable. This paper shows that CIL is learnable. Based on the theory, a new CIL algorithm is also proposed. Experimental results demonstrate its effectiveness.