# TinySubNets: An efficient and low capacity continual learning strategy

arXiv: https://arxiv.org/abs/2412.10869 · HTML: https://mlsanity.com/paper/2412.10869/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 27 references that are in the map.
- First posted on arXiv 14 Dec 2024.

## Builds on (most cited first, 16 of 16)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Green AI](https://mlsanity.com/paper/1907.10597/) (2019)
- [Gradient Projection Memory for Continual Learning](https://mlsanity.com/paper/2103.09762/) (2021)
- [Supermasks in Superposition](https://mlsanity.com/paper/2006.14769/) (2020)
- [RanPAC: Random Projections and Pre-trained Models for Continual Learning](https://mlsanity.com/paper/2307.02251/) (2023)
- [Adversarial Continual Learning](https://mlsanity.com/paper/2003.09553/) (2020)
- [Don't forget, there is more than forgetting: new metrics for Continual Learning](https://mlsanity.com/paper/1810.13166/) (2018)
- [Rehearsal-Free Continual Learning over Small Non-I.I.D. Batches](https://mlsanity.com/paper/1907.03799/) (2019)
- [Continual Learning with Scaled Gradient Projection](https://mlsanity.com/paper/2302.01386/) (2023)
- [NPCL: Neural Processes for Uncertainty-Aware Continual Learning](https://mlsanity.com/paper/2310.19272/) (2023)

## Built on by (most cited first, 1 of 1)

- [TSN-Affinity: Similarity-Driven Parameter Reuse for Continual Offline Reinforcement Learning](https://mlsanity.com/paper/2604.25898/) (2026)

## Abstract (arXiv)

Continual Learning (CL) is a highly relevant setting gaining traction in recent machine learning research. Among CL works, architectural and hybrid strategies are particularly effective due to their potential to adapt the model architecture as new tasks are presented. However, many existing solutions do not efficiently exploit model sparsity, and are prone to capacity saturation due to their inefficient use of available weights, which limits the number of learnable tasks. In this paper, we propose TinySubNets (TSN), a novel architectural CL strategy that addresses the issues through the unique combination of pruning with different sparsity levels, adaptive quantization, and weight sharing. Pruning identifies a subset of weights that preserve model performance, making less relevant weights available for future tasks. Adaptive quantization allows a single weight to be separated into multiple parts which can be assigned to different tasks. Weight sharing between tasks boosts the exploitation of capacity and task similarity, allowing for the identification of a better trade-off between model accuracy and capacity. These features allow TSN to efficiently leverage the available capacity, enhance knowledge transfer, and reduce computational resource consumption. Experimental results involving common benchmark CL datasets and scenarios show that our proposed strategy achieves better results in terms of accuracy than existing state-of-the-art CL strategies. Moreover, our strategy is shown to provide a significantly improved model capacity exploitation. Code released at: https://github.com/lifelonglab/tinysubnets.