# Keep and Learn: Continual Learning by Constraining the Latent Space for Knowledge Preservation in Neural Networks

arXiv: https://arxiv.org/abs/1805.10784 · HTML: https://mlsanity.com/paper/1805.10784/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 May 2018.

## Built on by (most cited first, 9 of 9)

- [Deep Learning for Chest X-ray Analysis: A Survey](https://mlsanity.com/paper/2103.08700/) (2021)
- [Learning Federated Visual Prompt in Null Space for MRI Reconstruction](https://mlsanity.com/paper/2303.16181/) (2023)
- [Continual Learning via Sequential Function-Space Variational Inference](https://mlsanity.com/paper/2312.17210/) (2023)
- [Continual Learning with Bayesian Model based on a Fixed Pre-trained Feature Extractor](https://mlsanity.com/paper/2204.13349/) (2022)
- [Preserving Earlier Knowledge in Continual Learning with the Help of All Previous Feature Extractors](https://mlsanity.com/paper/2104.13614/) (2021)
- [Continual Learning in Medical Imaging: A Survey and Practical Analysis](https://mlsanity.com/paper/2405.13482/) (2024)
- [Precision-Weighted Federated Learning](https://mlsanity.com/paper/2107.09627/) (2021)
- [Evaluation of Regularization-based Continual Learning Approaches: Application to HAR](https://mlsanity.com/paper/2304.13327/) (2023)
- [Discriminative Distillation to Reduce Class Confusion in Continual Learning](https://mlsanity.com/paper/2108.05187/) (2021)

## Abstract (arXiv)

Data is one of the most important factors in machine learning. However, even if we have high-quality data, there is a situation in which access to the data is restricted. For example, access to the medical data from outside is strictly limited due to the privacy issues. In this case, we have to learn a model sequentially only with the data accessible in the corresponding stage. In this work, we propose a new method for preserving learned knowledge by modeling the high-level feature space and the output space to be mutually informative, and constraining feature vectors to lie in the modeled space during training. The proposed method is easy to implement as it can be applied by simply adding a reconstruction loss to an objective function. We evaluate the proposed method on CIFAR-10/100 and a chest X-ray dataset, and show benefits in terms of knowledge preservation compared to previous approaches.