mlsanityData to 7 Oct 2026

Continual Learning for Text Classification with Information Disentanglement Based Regularization

arXiv 2104.05489 ↗First posted 12 Apr 2021cs.CL cs.AIcited by 86

Not on the map. Why?

Abstract (arXiv)

Continual learning has become increasingly important as it enables NLP models to constantly learn and gain knowledge over time. Previous continual learning methods are mainly designed to preserve knowledge from previous tasks, without much emphasis on how to well generalize models to new tasks. In this work, we propose an information disentanglement based regularization method for continual learning on text classification. Our proposed method first disentangles text hidden spaces into representations that are generic to all tasks and representations specific to each individual task, and further regularizes these representations differently to better constrain the knowledge required to generalize. We also introduce two simple auxiliary tasks: next sentence prediction and task-id prediction, for learning better generic and specific representation spaces. Experiments conducted on large-scale benchmarks demonstrate the effectiveness of our method in continual text classification tasks with various sequences and lengths over state-of-the-art baselines. We have publicly released our code at https://github.com/GT-SALT/IDBR.

Key facts. Cited by 86 papers in the map (data through 7 Oct 2026). Builds on 33 of its 49 references that are in the map. First posted on arXiv 12 Apr 2021.

Builds on

33 of 49 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  4. 2016Learning without Forgettingcited by 2,465
  5. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  7. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  8. 2016Progressive Neural Networkscited by 1,332
  9. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  10. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  11. 2017Continual Learning with Deep Generative Replaycited by 1,152
  12. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
21 more
  1. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  2. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  3. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  4. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  5. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  6. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 511
  7. 2017Toward Controlled Generation of Textcited by 271
  8. 2019Meta-Learning Representations for Continual Learningcited by 189
  9. 2019Episodic Memory in Lifelong Language Learningcited by 188
  10. 2020MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classificationcited by 186
  11. 2019LAMOL: LAnguage MOdeling for Lifelong Language Learningcited by 156
  12. 2020Continual Lifelong Learning in Natural Language Processing: A Surveycited by 152
  13. 2017Style Transfer in Text: Exploration and Evaluationcited by 145
  14. 2018Disentangled Representation Learning for Non-Parallel Text Style Transfercited by 127
  15. 2017Encoder Based Lifelong Learningcited by 102
  16. 2019Learning and Evaluating General Linguistic Intelligencecited by 78
  17. 2019Sentence Embedding Alignment for Lifelong Relation Extractioncited by 70
  18. 2019Generating Sentences from Disentangled Syntactic and Semantic Spacescited by 59
  19. 2020Efficient Meta Lifelong-Learning with Limited Memorycited by 48
  20. 2018Adversarial Decomposition of Text Representationcited by 18
  21. 2020Meta-Learning with Sparse Experience Replay for Lifelong Language Learningcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 14; 2023: 21; 2024: 25; 2025: 14; 2026: 7.

86 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  2. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 296
  3. 2023Progressive Prompts: Continual Learning for Language Modelscited by 195
  4. 2023Continual Pre-training of Language Modelscited by 194
  5. 2023Mitigating the Alignment Tax of RLHFcited by 181
  6. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 114
  7. 2024Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsalcited by 114
  8. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  9. 2021Time Waits for No One! Analysis and Challenges of Temporal Misalignmentcited by 91
  10. 2022Continual Learning of Natural Language Processing Tasks: A Surveycited by 83
  11. 2023Lifelong Language Pretraining with Distribution-Specialized Expertscited by 64
  12. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 60
38 more
  1. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  2. 2022Memory Efficient Continual Learning with Transformerscited by 43
  3. 2021Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learningcited by 35
  4. 2022Symbolic Replay: Scene Graph as Prompt for Continual Learning on VQA Taskcited by 35
  5. 2022Continual Training of Language Models for Few-Shot Learningcited by 33
  6. 2023CITB: A Benchmark for Continual Instruction Tuningcited by 32
  7. 2022Continual Sequence Generation with Adaptive Compositional Modulescited by 31
  8. 2024Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMscited by 31
  9. 2022Meta-attention for ViT-backed Continual Learningcited by 29
  10. 2024Scalable Language Model with Generalized Continual Learningcited by 28
  11. 2024Rehearsal-Free Modular and Compositional Continual Learning for Language Modelscited by 26
  12. 2023FedET: A Communication-Efficient Federated Class-Incremental Learning Framework Based on Enhanced Transformercited by 23
  13. 2022On Continual Model Refinement in Out-of-Distribution Data Streamscited by 22
  14. 2024Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Modelscited by 22
  15. 2021Prototype-Guided Memory Replay for Continual Learningcited by 19
  16. 2023Investigating Forgetting in Pre-Trained Representations Through Continual Learningcited by 19
  17. 2022Continual Learning with Transformers for Image Classificationcited by 17
  18. 2025Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuningcited by 16
  19. 2023GRASP: A Rehearsal Policy for Efficient Online Continual Learningcited by 13
  20. 2023Learn or Recall? Revisiting Incremental Learning with Pre-trained Language Modelscited by 13
  21. 2024MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuningcited by 13
  22. 2024Continual Learning Using a Kernel-Based Method Over Foundation Modelscited by 13
  23. 2024SurgeryV2: Bridging the Gap Between Model Merging and Multi-Task Learning with Deep Representation Surgerycited by 12
  24. 2023Class-Incremental Learning based on Label Generationcited by 11
  25. 2024ChronosLex: Time-aware Incremental Training for Temporal Generalization of Legal Classification Taskscited by 11
  26. 2023Preventing Catastrophic Forgetting in Continual Learning of New Natural Language Taskscited by 10
  27. 2024Q-Tuning: Queue-based Prompt Tuning for Lifelong Few-shot Language Learningcited by 9
  28. 2025Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methodscited by 9
  29. 2023Towards Anytime Fine-tuning: Continually Pre-trained Language Models with Hypernetwork Promptcited by 8
  30. 2024Self-Data Distillation for Recovering Quality in Pruned Large Language Modelscited by 8
  31. 2023InfoCL: Alleviating Catastrophic Forgetting in Continual Text Classification from An Information Theoretic Perspectivecited by 7
  32. 2023Sub-network Discovery and Soft-masking for Continual Learning of Mixed Taskscited by 7
  33. 2024In-context Continual Learning Assisted by an External Continual Learnercited by 7
  34. 2023Continual Learning From a Stream of APIscited by 6
  35. 2024Towards Practical Tool Usage for Continually Learning LLMscited by 6
  36. 2024MoE-CT: A Novel Approach For Large Language Models Training With Resistance To Catastrophic Forgettingcited by 6
  37. 2026Safety Alignment as Continual Learning: Mitigating the Alignment Tax via Orthogonal Gradient Projectioncited by 6
  38. 2022Exclusive Supermask Subnetwork Training for Continual Learningcited by 5

50 most cited of 86 All 86 (JSON)