mlsanityData to 14 Oct 2026

Lifelong Teacher-Student Network Learning

arXiv 2107.04689 ↗First posted 9 Jul 2021cs.LG cs.AI cs.CVcited by 18

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

A unique cognitive capability of humans consists in their ability to acquire new knowledge and skills from a sequence of experiences. Meanwhile, artificial intelligence systems are good at learning only the last given task without being able to remember the databases learnt in the past. We propose a novel lifelong learning methodology by employing a Teacher-Student network framework. While the Student module is trained with a new given database, the Teacher module would remind the Student about the information learnt in the past. The Teacher, implemented by a Generative Adversarial Network (GAN), is trained to preserve and replay past knowledge corresponding to the probabilistic representations of previously learn databases. Meanwhile, the Student module is implemented by a Variational Autoencoder (VAE) which infers its latent variable representation from both the output of the Teacher module as well as from the newly available database. Moreover, the Student module is trained to capture both continuous and discrete underlying data representations across different domains. The proposed lifelong learning framework is applied in supervised, semi-supervised and unsupervised training. The code is available~: urlhttps://github.com/dtuzi123/Lifelong-Teacher-Student-Network-Learning

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 40 of its 69 references that are in the map. First posted on arXiv 9 Jul 2021.

Builds on

40 of 69 references.

References · 40

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  5. 2013Auto-Encoding Variational Bayescited by 4,053
  6. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  7. 2016Improved Techniques for Training GANscited by 2,970
  8. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  9. 2016Learning without Forgettingcited by 2,496
  10. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  11. 2016Progressive Neural Networkscited by 1,361
  12. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
28 more
  1. 2017Continual Learning with Deep Generative Replaycited by 1,158
  2. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 860
  3. 2018Disentangling by Factorisingcited by 616
  4. 2019Three scenarios for continual learningcited by 614
  5. 2014Semi-Supervised Learning with Deep Generative Modelscited by 496
  6. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 372
  7. 2018Understanding disentangling in β-VAEcited by 335
  8. 2017VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learningcited by 200
  9. 2018Isolating Sources of Disentanglement in Variational Autoencoderscited by 167
  10. 2014A* Samplingcited by 151
  11. 2019Continual Unsupervised Representation Learningcited by 147
  12. 2019DIVA: Domain Invariant Variational Autoencoderscited by 115
  13. 2018An introduction to domain adaptation and transfer learningcited by 112
  14. 2020A survey on domain adaptation theory: learning bounds and theoretical guaranteescited by 109
  15. 2017Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networkscited by 107
  16. 2016Less-forgetting Learning in Deep Neural Networkscited by 97
  17. 2018Memory Replay GANs: learning to generate images from new categories without forgettingcited by 77
  18. 2017Learning Disentangled Representations with Semi-Supervised Deep Generative Modelscited by 72
  19. 2016AdaNet: Adaptive Structural Learning of Artificial Neural Networkscited by 54
  20. 2017Continual Learning in Generative Adversarial Netscited by 51
  21. 2017Lifelong Generative Modelingcited by 51
  22. 2018Learning Latent Subspaces in Variational Autoencoderscited by 51
  23. 2018Life-Long Disentangled Representation Learning with Cross-Domain Latent Homologiescited by 43
  24. 2017Rocket Launching: A Universal and Efficient Framework for Training Well-performing Light Netcited by 27
  25. 2018Auto-Encoding Total Correlation Explanationcited by 17
  26. 2017Adversarial Symmetric Variational Autoencodercited by 11
  27. 2019Defining Image Memorability using the Visual Memory Schemacited by 10
  28. 2017Adversarial Information Factorizationcited by 6

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 7; 2023: 1; 2024: 5; 2025: 3.

18 papers.

Sort:

By area:

  1. 2022Task-Free Continual Learning via Online Discrepancy Distance Learningcited by 22
  2. 2022Attention Distillation: self-supervised vision transformer students need more guidancecited by 16
  3. 2022Continual Variational Autoencoder Learning via Online Cooperative Memorizationcited by 12
  4. 2022Multi-Label Continual Learning using Augmented Graph Convolutional Networkcited by 12
  5. 2021Lifelong Infinite Mixture Model Based on Knowledge-Driven Dirichlet Processcited by 7
  6. 2022Online Unsupervised Domain Adaptation for Person Re-identificationcited by 7
  7. 2024Confidence Self-Calibration for Multi-Label Class-Incremental Learningcited by 6
  8. 2021Lifelong Generative Modelling Using Dynamic Expansion Graph Modelcited by 4
  9. 2023Meta-Learning Strategies through Value Maximization in Neural Networkscited by 4
  10. 2024Graph Memory Learning: Imitating Lifelong Remembering and Forgetting of Brain Networkscited by 4
  11. 2022Learning an evolved mixture model for task-free continual learningcited by 3
  12. 2024Source-Guided Similarity Preservation for Online Person Re-Identificationcited by 3
6 more
  1. 2025Neuroplasticity in Artificial Intelligence -- An Overview and Inspirations on Drop In & Out Learningcited by 3
  2. 2022Lifelong Generative Learning via Knowledge Reconstructioncited by 2
  3. 2024Collaborative Knowledge Distillation via a Learning-by-Education Node Communitycited by 1
  4. 2024Multi-Exposure Image Fusion via Distilled 3D LUT Grid with Editable Modecited by 0
  5. 2025CCD: Continual Consistency Diffusion for Lifelong Generative Modelingcited by 0
  6. 2025Unsupervised Video Continual Learning via Non-Parametric Deep Embedded Clusteringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Lifelong Twin Generative Adversarial Networks
  2. 2021Continual Learning in the Teacher-Student Setup: Impact of Task Similarity
  3. 2024Hybrid Data-Free Knowledge Distillation
  4. 2024CFTS-GAN: Continual Few-Shot Teacher Student for Generative Adversarial Networks
  5. 2023Teacher-Student Architecture for Knowledge Distillation: A Survey
  6. 2022Teacher-Student Architecture for Knowledge Learning: A Survey
  7. 2022Excess Risk of Two-Layer ReLU Neural Networks in Teacher-Student Settings and its Superiority to Kernel Methods
  8. 2023How a student becomes a teacher: learning and forgetting through Spectral methods