mlsanityData to 7 Oct 2026

Lifelong Mixture of Variational Autoencoders

arXiv 2107.04694 ↗First posted 9 Jul 2021cs.LG cs.AI cs.CVcited by 10

On the map: Mixture-of-experts → Mixture of LoRA experts

Abstract (arXiv)

In this paper, we propose an end-to-end lifelong learning mixture of experts. Each expert is implemented by a Variational Autoencoder (VAE). The experts in the mixture system are jointly trained by maximizing a mixture of individual component evidence lower bounds (MELBO) on the log-likelihood of the given training samples. The mixing coefficients in the mixture, control the contributions of each expert in the goal representation. These are sampled from a Dirichlet distribution whose parameters are determined through non-parametric estimation during lifelong learning. The model can learn new tasks fast when these are similar to those previously learnt. The proposed Lifelong mixture of VAE (L-MVAE) expands its architecture with new components when learning a completely new task. After the training, our model can automatically determine the relevant expert to be used when fed with new data samples. This mechanism benefits both the memory efficiency and the required computational cost as only one expert is used during the inference. The L-MVAE inference model is able to perform interpolation in the joint latent space across the data domains associated with different tasks and is shown to be efficient for disentangled learning representation.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts. Builds on 40 of its 69 references that are in the map. First posted on arXiv 9 Jul 2021.

Builds on

40 of 69 references.

References · 40

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  3. 2013Auto-Encoding Variational Bayescited by 3,980
  4. 2016Improved Techniques for Training GANscited by 2,938
  5. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  6. 2016Learning without Forgettingcited by 2,465
  7. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  8. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  9. 2016Progressive Neural Networkscited by 1,332
  10. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  11. 2017Continual Learning with Deep Generative Replaycited by 1,152
  12. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
28 more
  1. 2019Three scenarios for continual learningcited by 610
  2. 2019Gradient based sample selection for online continual learningcited by 568
  3. 2014Semi-Supervised Learning with Deep Generative Modelscited by 494
  4. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
  5. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 369
  6. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 368
  7. 2017Variational Continual Learningcited by 335
  8. 2018Understanding disentangling in β-VAEcited by 332
  9. 2016Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoderscited by 219
  10. 2016Adversarially Learned Inferencecited by 216
  11. 2018Multimodal Generative Models for Scalable Weakly-Supervised Learningcited by 200
  12. 2018Online Structured Laplace Approximations For Overcoming Catastrophic Forgettingcited by 197
  13. 2019Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Modelscited by 177
  14. 2017Generalization and Equilibrium in Generative Adversarial Nets (GANs)cited by 165
  15. 2019Continual Unsupervised Representation Learningcited by 147
  16. 2016Less-forgetting Learning in Deep Neural Networkscited by 97
  17. 2018Learning Disentangled Joint Continuous and Discrete Representationscited by 83
  18. 2019Lifelong GAN: Continual Learning for Conditional Image Generationcited by 78
  19. 2020Improved Techniques for Training Single-Image GANscited by 77
  20. 2017Learning Disentangled Representations with Semi-Supervised Deep Generative Modelscited by 72
  21. 2016AdaNet: Adaptive Structural Learning of Artificial Neural Networkscited by 54
  22. 2017Lifelong Generative Modelingcited by 51
  23. 2018Life-Long Disentangled Representation Learning with Cross-Domain Latent Homologiescited by 43
  24. 2020Learning latent representations across multiple data domains using Lifelong VAEGANcited by 35
  25. 2018Auto-Encoding Total Correlation Explanationcited by 17
  26. 2017Symmetric Variational Autoencoder and Connections to Adversarial Learningcited by 14
  27. 2017Adversarial Symmetric Variational Autoencodercited by 11
  28. 2018Multi-Source Neural Variational Inferencecited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 4; 2023: 2; 2024: 0; 2025: 1; 2026: 1.

10 papers.

  1. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
  2. 2022Verifying the Union of Manifolds Hypothesis for Image Datacited by 62
  3. 2022Task-Free Continual Learning via Online Discrepancy Distance Learningcited by 22
  4. 2025Mixture of Experts (MoE): A Big Data Perspectivecited by 17
  5. 2022Continual Variational Autoencoder Learning via Online Cooperative Memorizationcited by 12
  6. 2021Lifelong Infinite Mixture Model Based on Knowledge-Driven Dirichlet Processcited by 7
  7. 2021Lifelong Generative Modelling Using Dynamic Expansion Graph Modelcited by 4
  8. 2022Learning an evolved mixture model for task-free continual learningcited by 3
  9. 2023Bayesian Personalized Federated Learning with Shared and Personalized Uncertainty Representationscited by 2
  10. 2026Modular Continual Learning via Zero-Leakage Reconstruction Routing and Autonomous Task Discoverycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2019Mixture-of-Experts Variational Autoencoder for Clustering and Generating from Similarity-Based Representations on Single Cell Data
  2. 2025Aggregation of Dependent Expert Distributions in Multimodal Variational Autoencoders
  3. 2023Learning multi-modal generative models with permutation-invariant encoders and tighter variational objectives
  4. 2025LLaVA-CMoE: Towards Continual Mixture of Experts for Large Vision-Language Models
  5. 2025Beyond Redundancy: Diverse and Specialized Multi-Expert Sparse Autoencoder
  6. 2025CoVAE: Consistency Training of Variational Autoencoders
  7. 2024Unity by Diversity: Improved Representation Learning in Multimodal VAEs
  8. 2026Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees