# Multiband VAE: Latent Space Alignment for Knowledge Consolidation in Continual Learning

arXiv: https://arxiv.org/abs/2106.12196 · HTML: https://mlsanity.com/paper/2106.12196/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 23 Jun 2021.

## Builds on (most cited first, 22 of 22)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [Three scenarios for continual learning](https://mlsanity.com/paper/1904.07734/) (2019)
- [Assessing Generative Models via Precision and Recall](https://mlsanity.com/paper/1806.00035/) (2018)
- [Variational Continual Learning](https://mlsanity.com/paper/1710.10628/) (2017)
- [Continual learning with hypernetworks](https://mlsanity.com/paper/1906.00695/) (2019)
- [VAE with a VampPrior](https://mlsanity.com/paper/1705.07120/) (2017)
- [Continual Unsupervised Representation Learning](https://mlsanity.com/paper/1910.14481/) (2019)
- [Generative Models from the perspective of Continual Learning](https://mlsanity.com/paper/1812.09111/) (2018)
- [Lifelong Generative Modeling](https://mlsanity.com/paper/1705.09847/) (2017)
- [Particle Cloud Generation with Message Passing Generative Adversarial Networks](https://mlsanity.com/paper/2106.11535/) (2021)
- [Life-Long Disentangled Representation Learning with Cross-Domain Latent Homologies](https://mlsanity.com/paper/1808.06508/) (2018)
- [Learning latent representations across multiple data domains using Lifelong VAEGAN](https://mlsanity.com/paper/2007.10221/) (2020)
- [BooVAE: Boosting Approach for Continual Learning of VAE](https://mlsanity.com/paper/1908.11853/) (2019)
- [End-to-end Sinkhorn Autoencoder with Noise Generator](https://mlsanity.com/paper/2006.06704/) (2020)

## Built on by (most cited first, 5 of 5)

- [Exploring Continual Learning of Diffusion Models](https://mlsanity.com/paper/2303.15342/) (2023)
- [Joint Diffusion models in Continual Learning](https://mlsanity.com/paper/2411.08224/) (2024)
- [C2VAE: Gaussian Copula-based VAE Differing Disentangled from Coupled Representations with Contrastive Posterior](https://mlsanity.com/paper/2309.13303/) (2023)
- [Adapt & Align: Continual Learning with Generative Models Latent Space Alignment](https://mlsanity.com/paper/2312.13699/) (2023)
- [Continual Learning in Modern Hopfield Networks with an Application to Diffusion Models](https://mlsanity.com/paper/2605.27975/) (2026)

## Abstract (arXiv)

We propose a new method for unsupervised generative continual learning through realignment of Variational Autoencoder's latent space. Deep generative models suffer from catastrophic forgetting in the same way as other neural structures. Recent generative continual learning works approach this problem and try to learn from new data without forgetting previous knowledge. However, those methods usually focus on artificial scenarios where examples share almost no similarity between subsequent portions of data - an assumption not realistic in the real-life applications of continual learning. In this work, we identify this limitation and posit the goal of generative continual learning as a knowledge accumulation task. We solve it by continuously aligning latent representations of new data that we call bands in additional latent space where examples are encoded independently of their source task. In addition, we introduce a method for controlled forgetting of past data that simplifies this process. On top of the standard continual learning benchmarks, we propose a novel challenging knowledge consolidation scenario and show that the proposed approach outperforms state-of-the-art by up to twofold across all experiments and the additional real-life evaluation. To our knowledge, Multiband VAE is the first method to show forward and backward knowledge transfer in generative continual learning.