mlsanityData to 7 Oct 2026

Lifelong Language Knowledge Distillation

arXiv 2010.02123 ↗First posted 5 Oct 2020cs.CL cs.AI cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

It is challenging to perform lifelong language learning (LLL) on a stream of different tasks without any performance degradation comparing to the multi-task counterparts. To address this issue, we present Lifelong Language Knowledge Distillation (L2KD), a simple but efficient method that can be easily applied to existing LLL architectures in order to mitigate the degradation. Specifically, when the LLL model is trained on a new task, we assign a teacher model to first learn the new task, and pass the knowledge to the LLL model via knowledge distillation. Therefore, the LLL model can better adapt to the new task while keeping the previously learned knowledge. Experiments show that the proposed L2KD consistently improves previous state-of-the-art models, and the degradation comparing to multi-task models in LLL tasks is well mitigated for both sequence generation and text classification tasks.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 11; 2023: 9; 2024: 4; 2025: 1; 2026: 1.

31 papers.

Sort:

  1. 2023Continual Pre-training of Language Modelscited by 194
  2. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 114
  3. 2021LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5cited by 103
  4. 2021Achieving Forgetting Prevention and Knowledge Transfer in Continual Learningcited by 91
  5. 2022Memory Efficient Continual Learning with Transformerscited by 43
  6. 2022Continual Training of Language Models for Few-Shot Learningcited by 33
  7. 2022Continual Sequence Generation with Adaptive Compositional Modulescited by 31
  8. 2022Incremental Prompting: Episodic Memory Prompt for Lifelong Event Detectioncited by 23
  9. 2021Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updatescited by 13
  10. 2023Learn or Recall? Revisiting Incremental Learning with Pre-trained Language Modelscited by 13
  11. 2023Lifelong Sequence Generation with Dynamic Module Expansion and Adaptationcited by 10
  12. 2024A Comprehensive Review of Knowledge Distillation in Computer Visioncited by 10
19 more
  1. 2023Teamwork Is Not Always Good: An Empirical Study of Classifier Drift in Class-incremental Information Extractioncited by 9
  2. 2022Prompt Conditioned VAE: Enhancing Generative Replay for Lifelong Learning in Task-Oriented Dialoguecited by 8
  3. 2023Towards Anytime Fine-tuning: Continually Pre-trained Language Models with Hypernetwork Promptcited by 8
  4. 2023Continual Learning with Dirichlet Generative-based Rehearsalcited by 7
  5. 2024In-context Continual Learning Assisted by an External Continual Learnercited by 7
  6. 2023Improving Prediction Backward-Compatiblility in NLP Model Upgrade with Gated Fusioncited by 6
  7. 2022Enabling All In-Edge Deep Learning: A Literature Reviewcited by 4
  8. 2022RVAE-LAMOL: Residual Variational Autoencoder to Enhance Lifelong Language Learningcited by 4
  9. 2022Continual Knowledge Distillation for Neural Machine Translationcited by 4
  10. 2023Schema-adaptable Knowledge Graph Constructioncited by 4
  11. 2023Knowledge Distillation of LLM for Automatic Scoring of Science Education Assessmentscited by 4
  12. 2021Reminding the Incremental Language Model via Data-Free Self-Distillationcited by 3
  13. 2024HOP to the Next Tasks and Domains for Continual Learning in NLPcited by 3
  14. 2022BERT WEAVER: Using WEight AVERaging to enable lifelong learning for transformer-based models in biomedical semantic search enginescited by 2
  15. 2025Memorization Inheritance in Sequence-Level Knowledge Distillation for Neural Machine Translationcited by 2
  16. 2026KD4MT: A Survey of Knowledge Distillation for Machine Translationcited by 2
  17. 2022Semi-Supervised Lifelong Language Learningcited by 1
  18. 2024Can LLMs Learn New Concepts Incrementally without Forgetting?cited by 1
  19. 2022Ask Question First for Enhancing Lifelong Language Learningcited by 0