# Efficient Continual Learning in Neural Networks with Embedding Regularization

arXiv: https://arxiv.org/abs/1909.03742 · HTML: https://mlsanity.com/paper/1909.03742/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Sep 2019.

## Built on by (most cited first, 20 of 20)

- [Unsupervised Continual Learning Via Pseudo Labels](https://mlsanity.com/paper/2104.07164/) (2021)
- [Task-agnostic Continual Learning with Hybrid Probabilistic Models](https://mlsanity.com/paper/2106.12772/) (2021)
- [Practical Recommendations for Replay-based Continual Learning Methods](https://mlsanity.com/paper/2203.10317/) (2022)
- [Online Continual Learning Via Candidates Voting](https://mlsanity.com/paper/2110.08855/) (2021)
- [Unveiling the Anomalies in an Ever-Changing World: A Benchmark for Pixel-Level Anomaly Detection in Continual Learning](https://mlsanity.com/paper/2403.15463/) (2024)
- [GROWN: GRow Only When Necessary for Continual Learning](https://mlsanity.com/paper/2110.00908/) (2021)
- [Continual Learning Approaches for Anomaly Detection](https://mlsanity.com/paper/2212.11192/) (2022)
- [Towards Continuous Reuse of Graph Models via Holistic Memory Diversification](https://mlsanity.com/paper/2406.07413/) (2024)
- [Online Continual Learning via the Meta-learning Update with Multi-scale Knowledge Distillation and Data Augmentation](https://mlsanity.com/paper/2209.06107/) (2022)
- [Continual Learning with Invertible Generative Models](https://mlsanity.com/paper/2202.05694/) (2022)
- [Learning to Classify New Foods Incrementally Via Compressed Exemplars](https://mlsanity.com/paper/2404.07507/) (2024)
- [Smaller Is Better: An Analysis of Instance Quantity/Quality Trade-off in Rehearsal-based Continual Learning](https://mlsanity.com/paper/2105.14106/) (2021)
- [Centroids Matching: an efficient Continual Learning approach operating in the embedding space](https://mlsanity.com/paper/2208.02048/) (2022)
- [DILF-EN framework for Class-Incremental Learning](https://mlsanity.com/paper/2112.12385/) (2021)
- [Continual Learning for Steganalysis](https://mlsanity.com/paper/2209.01326/) (2022)
- [From Lab to Pocket: A Novel Continual Learning-based Mobile Application for Screening COVID-19](https://mlsanity.com/paper/2410.12589/) (2024)
- [UrbanMind: Towards Urban General Intelligence via Tool-Enhanced Retrieval-Augmented Generation and Multilevel Optimization](https://mlsanity.com/paper/2507.04706/) (2025)
- [Multi-Level Feature Fusion for Continual Learning in Visual Quality Inspection](https://mlsanity.com/paper/2601.00725/) (2026)
- [From Data Heterogeneity to Convergence: A Data-Centric Review of Federated Learning](https://mlsanity.com/paper/2606.10595/) (2026)
- [MoDiCoL: A Modular Diagnostic Continual Learning Dataset for Robust Speech Recognition](https://mlsanity.com/paper/2606.14459/) (2026)

## Abstract (arXiv)

Continual learning of deep neural networks is a key requirement for scaling them up to more complex applicative scenarios and for achieving real lifelong learning of these architectures. Previous approaches to the problem have considered either the progressive increase in the size of the networks, or have tried to regularize the network behavior to equalize it with respect to previously observed tasks. In the latter case, it is essential to understand what type of information best represents this past behavior. Common techniques include regularizing the past outputs, gradients, or individual weights. In this work, we propose a new, relatively simple and efficient method to perform continual learning by regularizing instead the network internal embeddings. To make the approach scalable, we also propose a dynamic sampling strategy to reduce the memory footprint of the required external storage. We show that our method performs favorably with respect to state-of-the-art approaches in the literature, while requiring significantly less space in memory and computational time. In addition, inspired inspired by to recent works, we evaluate the impact of selecting a more flexible model for the activation functions inside the network, evaluating the impact of catastrophic forgetting on the activation functions themselves.