# Self-paced Weight Consolidation for Continual Learning

arXiv: https://arxiv.org/abs/2307.10845 · HTML: https://mlsanity.com/paper/2307.10845/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 75 references that are in the map.
- First posted on arXiv 20 Jul 2023.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [Progress & Compress: A scalable framework for continual learning](https://mlsanity.com/paper/1805.06370/) (2018)
- [Expert Gate: Lifelong Learning with a Network of Experts](https://mlsanity.com/paper/1611.06194/) (2016)
- [Self-paced Contrastive Learning with Hybrid Memory for Domain Adaptive Object Re-ID](https://mlsanity.com/paper/2006.02713/) (2020)
- [Modeling the Background for Incremental Learning in Semantic Segmentation](https://mlsanity.com/paper/2002.00718/) (2020)
- [FearNet: Brain-Inspired Model for Incremental Learning](https://mlsanity.com/paper/1711.10563/) (2017)
- [PLOP: Learning without Forgetting for Continual Semantic Segmentation](https://mlsanity.com/paper/2011.11390/) (2020)
- [Incremental Learning Techniques for Semantic Segmentation](https://mlsanity.com/paper/1907.13372/) (2019)
- [Federated Class-Incremental Learning](https://mlsanity.com/paper/2203.11473/) (2022)
- [Class-Incremental Continual Learning into the eXtended DER-verse](https://mlsanity.com/paper/2201.00766/) (2022)
- [Encoder Based Lifelong Learning](https://mlsanity.com/paper/1704.01920/) (2017)
- [Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations](https://mlsanity.com/paper/2103.06342/) (2021)
- [Representation Compensation Networks for Continual Semantic Segmentation](https://mlsanity.com/paper/2203.05402/) (2022)
- [Continual Detection Transformer for Incremental Object Detection](https://mlsanity.com/paper/2304.03110/) (2023)
- [PIVOT: Prompting for Video Continual Learning](https://mlsanity.com/paper/2212.04842/) (2022)
- [Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation](https://mlsanity.com/paper/2203.14098/) (2022)
- [Federated Incremental Semantic Segmentation](https://mlsanity.com/paper/2304.04620/) (2023)
- [Continual Learning via Bit-Level Information Preserving](https://mlsanity.com/paper/2105.04444/) (2021)
- [ALIFE: Adaptive Logit Regularizer and Feature Replay for Incremental Semantic Segmentation](https://mlsanity.com/paper/2210.06816/) (2022)
- [Prototype-Guided Memory Replay for Continual Learning](https://mlsanity.com/paper/2108.12641/) (2021)
- [Self-Paced Multi-Task Learning](https://mlsanity.com/paper/1604.01474/) (2016)

## Built on by (most cited first, 9 of 9)

- [Local Feature Matching Using Deep Learning: A Survey](https://mlsanity.com/paper/2401.17592/) (2024)
- [A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Application](https://mlsanity.com/paper/2310.14277/) (2023)
- [MuseumMaker: Continual Style Customization without Catastrophic Forgetting](https://mlsanity.com/paper/2404.16612/) (2024)
- [Dynamic Transformer Architecture for Continual Learning of Multimodal Tasks](https://mlsanity.com/paper/2401.15275/) (2024)
- [Cs2K: Class-specific and Class-shared Knowledge Guidance for Incremental Semantic Segmentation](https://mlsanity.com/paper/2407.09047/) (2024)
- [Prompt Customization for Continual Learning](https://mlsanity.com/paper/2404.18060/) (2024)
- [Constructing Enhanced Mutual Information for Online Class-Incremental Learning](https://mlsanity.com/paper/2407.18526/) (2024)
- [Continual Learning-Based Unified Model for Unpaired Image Restoration Tasks](https://mlsanity.com/paper/2507.19184/) (2025)
- [PKI: Prior Knowledge-Infused Neural Network for Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2601.08493/) (2026)

## Abstract (arXiv)

Continual learning algorithms which keep the parameters of new tasks close to that of previous tasks, are popular in preventing catastrophic forgetting in sequential task learning settings. However, 1) the performance for the new continual learner will be degraded without distinguishing the contributions of previously learned tasks; 2) the computational cost will be greatly increased with the number of tasks, since most existing algorithms need to regularize all previous tasks when learning new tasks. To address the above challenges, we propose a self-paced Weight Consolidation (spWC) framework to attain robust continual learning via evaluating the discriminative contributions of previous tasks. To be specific, we develop a self-paced regularization to reflect the priorities of past tasks via measuring difficulty based on key performance indicator (i.e., accuracy). When encountering a new task, all previous tasks are sorted from "difficult" to "easy" based on the priorities. Then the parameters of the new continual learner will be learned via selectively maintaining the knowledge amongst more difficult past tasks, which could well overcome catastrophic forgetting with less computational cost. We adopt an alternative convex search to iteratively update the model parameters and priority weights in the bi-convex formulation. The proposed spWC framework is plug-and-play, which is applicable to most continual learning algorithms (e.g., EWC, MAS and RCIL) in different directions (e.g., classification and segmentation). Experimental results on several public benchmark datasets demonstrate that our proposed framework can effectively improve performance when compared with other popular continual learning algorithms.