# Overcoming Multi-Model Forgetting

arXiv: https://arxiv.org/abs/1902.08232 · HTML: https://mlsanity.com/paper/1902.08232/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Feb 2019.

## Built on by (most cited first, 14 of 14)

- [iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients](https://mlsanity.com/paper/2106.10784/) (2021)
- [AlphaNet: Improved Training of Supernets with Alpha-Divergence](https://mlsanity.com/paper/2102.07954/) (2021)
- [Pi-NAS: Improving Neural Architecture Search by Reducing Supernet Training Consistency Shift](https://mlsanity.com/paper/2108.09671/) (2021)
- [A Survey on Computationally Efficient Neural Architecture Search](https://mlsanity.com/paper/2206.01520/) (2022)
- [Landmark Regularization: Ranking Guided Super-Net Training in Neural Architecture Search](https://mlsanity.com/paper/2104.05309/) (2021)
- [CLOSE: Curriculum Learning On the Sharing Extent Towards Better One-shot NAS](https://mlsanity.com/paper/2207.07868/) (2022)
- [Efficient Automation of Neural Network Design: A Survey on Differentiable Neural Architecture Search](https://mlsanity.com/paper/2304.05405/) (2023)
- [Automated Dominative Subspace Mining for Efficient Neural Architecture Search](https://mlsanity.com/paper/2210.17180/) (2022)
- [Enhancing Once-For-All: A Study on Parallel Blocks, Skip Connections and Early Exits](https://mlsanity.com/paper/2302.01888/) (2023)
- [Neural Architecture Transfer 2: A Paradigm for Improving Efficiency in Multi-Objective Neural Architecture Search](https://mlsanity.com/paper/2307.00960/) (2023)
- [Brain-inspired Evolutionary Architectures for Spiking Neural Networks](https://mlsanity.com/paper/2309.05263/) (2023)
- [TS-ENAS:Two-Stage Evolution for Cell-based Network Architecture Search](https://mlsanity.com/paper/2310.09525/) (2023)
- [Multiple Population Alternate Evolution Neural Architecture Search](https://mlsanity.com/paper/2403.07035/) (2024)
- [Multi-Task Neural Architecture Search Using Architecture Embedding and Transfer Rank](https://mlsanity.com/paper/2504.00772/) (2025)

## Abstract (arXiv)

We identify a phenomenon, which we refer to as multi-model forgetting, that occurs when sequentially training multiple deep networks with partially-shared parameters; the performance of previously-trained models degrades as one optimizes a subsequent one, due to the overwriting of shared parameters. To overcome this, we introduce a statistically-justified weight plasticity loss that regularizes the learning of a model's shared parameters according to their importance for the previous models, and demonstrate its effectiveness when training two models sequentially and for neural architecture search. Adding weight plasticity in neural architecture search preserves the best models to the end of the search and yields improved results in both natural language processing and computer vision tasks.