# Lifelong Generative Modeling

arXiv: https://arxiv.org/abs/1705.09847 · HTML: https://mlsanity.com/paper/1705.09847/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 May 2017.

## Built on by (most cited first, 30 of 51)

- [A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning](https://mlsanity.com/paper/2307.09218/) (2023)
- [Recent Advances of Continual Learning in Computer Vision: An Overview](https://mlsanity.com/paper/2109.11369/) (2021)
- [An Efficient Domain-Incremental Learning Approach to Drive in All Weather Conditions](https://mlsanity.com/paper/2204.08817/) (2022)
- [NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks](https://mlsanity.com/paper/2206.09117/) (2022)
- [The Challenges of Continuous Self-Supervised Learning](https://mlsanity.com/paper/2203.12710/) (2022)
- [Lifelong Teacher-Student Network Learning](https://mlsanity.com/paper/2107.04689/) (2021)
- [SCALE: Online Self-Supervised Lifelong Learning without Prior Knowledge](https://mlsanity.com/paper/2208.11266/) (2022)
- [Unveiling the Anomalies in an Ever-Changing World: A Benchmark for Pixel-Level Anomaly Detection in Continual Learning](https://mlsanity.com/paper/2403.15463/) (2024)
- [Continual Variational Autoencoder Learning via Online Cooperative Memorization](https://mlsanity.com/paper/2207.10131/) (2022)
- [Continual Learning on Graphs: A Survey](https://mlsanity.com/paper/2402.06330/) (2024)
- [Beyond Supervised Continual Learning: a Review](https://mlsanity.com/paper/2208.14307/) (2022)
- [Lifelong Mixture of Variational Autoencoders](https://mlsanity.com/paper/2107.04694/) (2021)
- [IB-DRR: Incremental Learning with Information-Back Discrete Representation Replay](https://mlsanity.com/paper/2104.10588/) (2021)
- [Memory-efficient Reinforcement Learning with Value-based Knowledge Consolidation](https://mlsanity.com/paper/2205.10868/) (2022)
- [Continual Learning Approaches for Anomaly Detection](https://mlsanity.com/paper/2212.11192/) (2022)
- [Learning "O" Helps for Learning More: Handling the Concealed Entity Problem for Class-incremental NER](https://mlsanity.com/paper/2210.04676/) (2022)
- [Lifelong Infinite Mixture Model Based on Knowledge-Driven Dirichlet Process](https://mlsanity.com/paper/2108.12278/) (2021)
- [Knowledge Restore and Transfer for Multi-label Class-Incremental Learning](https://mlsanity.com/paper/2302.13334/) (2023)
- [Lifelong Twin Generative Adversarial Networks](https://mlsanity.com/paper/2107.04708/) (2021)
- [Multiband VAE: Latent Space Alignment for Knowledge Consolidation in Continual Learning](https://mlsanity.com/paper/2106.12196/) (2021)
- [CFA: Constraint-based Finetuning Approach for Generalized Few-Shot Object Detection](https://mlsanity.com/paper/2204.05220/) (2022)
- [GUIDE: Guidance-based Incremental Learning with Diffusion Models](https://mlsanity.com/paper/2403.03938/) (2024)
- [Lifelong Generative Modelling Using Dynamic Expansion Graph Model](https://mlsanity.com/paper/2112.08370/) (2021)
- [Multi-Stage Knowledge Integration of Vision-Language Models for Continual Learning](https://mlsanity.com/paper/2411.06764/) (2024)
- [Integral Continual Learning Along the Tangent Vector Field of Tasks](https://mlsanity.com/paper/2211.13108/) (2022)
- [Adiabatic replay for continual learning](https://mlsanity.com/paper/2303.13157/) (2023)
- [CEL: A Continual Learning Model for Disease Outbreak Prediction by Leveraging Domain Adaptation via Elastic Weight Consolidation](https://mlsanity.com/paper/2401.08940/) (2024)
- [Lifelong Generative Learning via Knowledge Reconstruction](https://mlsanity.com/paper/2201.06418/) (2022)
- [Transfer Learning via Test-Time Neural Networks Aggregation](https://mlsanity.com/paper/2206.13399/) (2022)
- [Class-Prototype Conditional Diffusion Model with Gradient Projection for Continual Learning](https://mlsanity.com/paper/2312.06710/) (2023)

## Abstract (arXiv)

Lifelong learning is the problem of learning multiple consecutive tasks in a sequential manner, where knowledge gained from previous tasks is retained and used to aid future learning over the lifetime of the learner. It is essential towards the development of intelligent machines that can adapt to their surroundings. In this work we focus on a lifelong learning approach to unsupervised generative modeling, where we continuously incorporate newly observed distributions into a learned model. We do so through a student-teacher Variational Autoencoder architecture which allows us to learn and preserve all the distributions seen so far, without the need to retain the past data nor the past models. Through the introduction of a novel cross-model regularizer, inspired by a Bayesian update rule, the student model leverages the information learned by the teacher, which acts as a probabilistic knowledge store. The regularizer reduces the effect of catastrophic interference that appears when we learn over sequences of distributions. We validate our model's performance on sequential variants of MNIST, FashionMNIST, PermutedMNIST, SVHN and Celeb-A and demonstrate that our model mitigates the effects of catastrophic interference faced by neural networks in sequential learning scenarios.