# LightCL: Compact Continual Learning with Low Memory Footprint For Edge Device

arXiv: https://arxiv.org/abs/2407.10545 · HTML: https://mlsanity.com/paper/2407.10545/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 32 references that are in the map.
- First posted on arXiv 15 Jul 2024.

## Builds on (most cited first, 23 of 23)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [On Tiny Episodic Memories in Continual Learning](https://mlsanity.com/paper/1902.10486/) (2019)
- [Co2L: Contrastive Continual Learning](https://mlsanity.com/paper/2106.14413/) (2021)
- [REMIND Your Neural Network to Prevent Catastrophic Forgetting](https://mlsanity.com/paper/1910.02509/) (2019)
- [Compressing Deep Convolutional Networks using Vector Quantization](https://mlsanity.com/paper/1412.6115/) (2014)
- [Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics](https://mlsanity.com/paper/2007.07400/) (2020)
- [On-Device Training Under 256KB Memory](https://mlsanity.com/paper/2206.15472/) (2022)
- [Measuring and regularizing networks in function space](https://mlsanity.com/paper/1805.08289/) (2018)
- [AI and Memory Wall](https://mlsanity.com/paper/2403.14123/) (2024)
- [A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning](https://mlsanity.com/paper/2307.09218/) (2023)
- [SparCL: Sparse Continual Learning on the Edge](https://mlsanity.com/paper/2209.09476/) (2022)
- [Tiny Machine Learning: Progress and Futures](https://mlsanity.com/paper/2403.19076/) (2024)
- [Learn-Prune-Share for Lifelong Learning](https://mlsanity.com/paper/2012.06956/) (2020)
- [EsaCL: Efficient Continual Learning of Sparse Models](https://mlsanity.com/paper/2401.05667/) (2024)

## Built on by (most cited first, 1 of 1)

- [SAMCL: Empowering SAM to Continually Learn from Dynamic Domains with Extreme Storage Efficiency](https://mlsanity.com/paper/2412.05012/) (2024)

## Abstract (arXiv)

Continual learning (CL) is a technique that enables neural networks to constantly adapt to their dynamic surroundings. Despite being overlooked for a long time, this technology can considerably address the customized needs of users in edge devices. Actually, most CL methods require huge resource consumption by the training behavior to acquire generalizability among all tasks for delaying forgetting regardless of edge scenarios. Therefore, this paper proposes a compact algorithm called LightCL, which evaluates and compresses the redundancy of already generalized components in structures of the neural network. Specifically, we consider two factors of generalizability, learning plasticity and memory stability, and design metrics of both to quantitatively assess generalizability of neural networks during CL. This evaluation shows that generalizability of different layers in a neural network exhibits a significant variation. Thus, we textitMaintain Generalizability by freezing generalized parts without the resource-intensive training process and textitMemorize Feature Patterns by stabilizing feature extracting of previous tasks to enhance generalizability for less-generalized parts with a little extra memory, which is far less than the reduction by freezing. Experiments illustrate that LightCL outperforms other state-of-the-art methods and reduces at most 6.16× memory footprint. We also verify the effectiveness of LightCL on the edge device.