mlsanityData to 14 Oct 2026

Class-Incremental Learning based on Label Generation

arXiv 2306.12619 ↗First posted 22 Jun 2023cs.CL cs.AI cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Despite the great success of pre-trained language models, it is still a challenge to use these models for continual learning, especially for the class-incremental learning (CIL) setting due to catastrophic forgetting (CF). This paper reports our finding that if we formulate CIL as a continual label generation problem, CF is drastically reduced and the generalizable representations of pre-trained models can be better retained. We thus propose a new CIL method (VAG) that also leverages the sparsity of vocabulary to focus the generation and creates pseudo-replay samples by using label semantics. Experimental results show that VAG outperforms baselines by a large margin.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 37 of its 71 references that are in the map. First posted on arXiv 22 Jun 2023.

Builds on

37 of 71 references.

References · 37

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  7. 2016Learning without Forgettingcited by 2,496
  8. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  9. 2017Continual Learning with Deep Generative Replaycited by 1,158
  10. 2021Learning to Prompt for Continual Learningcited by 944
  11. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 911
  12. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
25 more
  1. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 681
  2. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 433
  3. 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 386
  4. 2018FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluationcited by 254
  5. 2020Supermasks in Superpositioncited by 227
  6. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 227
  7. 2023Continual Pre-training of Language Modelscited by 198
  8. 2019LAMOL: LAnguage MOdeling for Lifelong Language Learningcited by 160
  9. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 157
  10. 2022Fine-tuned Language Models are Continual Learnerscited by 156
  11. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 132
  12. 2021LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5cited by 104
  13. 2021Continual Learning for Text Classification with Information Disentanglement Based Regularizationcited by 86
  14. 2022A Theoretical Study on Solving Continual Learningcited by 80
  15. 2020Continual Learning for Natural Language Generation in Task-oriented Dialog Systemscited by 47
  16. 2022Continual Prompt Tuning for Dialog State Trackingcited by 44
  17. 2022Continual Training of Language Models for Few-Shot Learningcited by 33
  18. 2022Continual Sequence Generation with Adaptive Compositional Modulescited by 31
  19. 2021CLASSIC: Continual and Contrastive Learning of Aspect Sentiment Classification Taskscited by 29
  20. 2022Consistent Representation Learning for Continual Relation Extractioncited by 28
  21. 2022ConTinTin: Continual Learning from Task Instructionscited by 26
  22. 2023Adapting a Language Model While Preserving its General Knowledgecited by 23
  23. 2021Lifelong Intent Detection via Multi-Strategy Rebalancingcited by 12
  24. 2023ACCENT: An Automatic Event Commonsense Evaluation Metric for Open-Domain Dialogue Systemscited by 11
  25. 2022FormLM: Recommending Creation Ideas for Online Forms by Modelling Semantic and Structural Informationcited by 3

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 8; 2025: 1.

11 papers.

  1. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
  2. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 64
  3. 2024Generative Multi-modal Models are Good Class-Incremental Learnerscited by 22
  4. 2023Class Incremental Learning via Likelihood Ratio Based Task Predictioncited by 19
  5. 2024Continual Learning Using a Kernel-Based Method Over Foundation Modelscited by 14
  6. 2023Learn or Recall? Revisiting Incremental Learning with Pre-trained Language Modelscited by 13
  7. 2024In-context Continual Learning Assisted by an External Continual Learnercited by 7
  8. 2024CLoG: Benchmarking Continual Learning of Image Generation Modelscited by 5
  9. 2024Continual Learning Using Only Large Language Model Promptingcited by 5
  10. 2025Achieving Upper Bound Accuracy of Joint Training in Continual Learningcited by 4
  11. 2024Can LLMs Learn New Concepts Incrementally without Forgetting?cited by 1