mlsanityData to 14 Oct 2026

CITB: A Benchmark for Continual Instruction Tuning

arXiv 2310.14510 ↗First posted 23 Oct 2023cs.CLcited by 33

Not on the map. Why?

Abstract (arXiv)

Continual learning (CL) is a paradigm that aims to replicate the human ability to learn and accumulate knowledge continually without forgetting previous knowledge and transferring it to new tasks. Recent instruction tuning (IT) involves fine-tuning models to make them more adaptable to solving NLP tasks in general. However, it is still uncertain how instruction tuning works in the context of CL tasks. This challenging yet practical problem is formulated as Continual Instruction Tuning (CIT). In this work, we establish a CIT benchmark consisting of learning and evaluation protocols. We curate two long dialogue task streams of different types, InstrDialog and InstrDialog++, to study various CL methods systematically. Our experiments show that existing CL methods do not effectively leverage the rich natural language instructions, and fine-tuning an instruction-tuned model sequentially can yield similar or better results. We further explore different aspects that might affect the learning of CIT. We hope this benchmark will facilitate more research in this direction.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 23 of its 30 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

23 of 30 references.

References · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  9. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  10. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  11. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  12. 2023LIMA: Less Is More for Alignmentcited by 1,100
11 more
  1. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
  2. 2022Fine-tuned Language Models are Continual Learnerscited by 156
  3. 2020Continual Lifelong Learning in Natural Language Processing: A Surveycited by 152
  4. 2019A continual learning survey: Defying forgetting in classification taskscited by 118
  5. 2021Achieving Forgetting Prevention and Knowledge Transfer in Continual Learningcited by 91
  6. 2021Continual Learning for Text Classification with Information Disentanglement Based Regularizationcited by 86
  7. 2020Continual Learning in Task-Oriented Dialogue Systemscited by 83
  8. 2022Continual Learning of Natural Language Processing Tasks: A Surveycited by 83
  9. 2023Augmented Large Language Models with Parametric Knowledge Guidingcited by 37
  10. 2022Continual Few-shot Relation Learning via Embedding Space Regularization and Data Augmentationcited by 17
  11. 2023TaskWeb: Selecting Better Source Tasks for Multi-task NLPcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 18; 2025: 6; 2026: 7.

33 papers.

Sort:

By area:

  1. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  2. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
  3. 2025From RAG to Memory: Non-Parametric Continual Learning for Large Language Modelscited by 231
  4. 2024Continual Learning for Large Language Models: A Surveycited by 147
  5. 2024Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsalcited by 118
  6. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 83
  7. 2024CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Modelcited by 61
  8. 2023Continual Instruction Tuning for Large Multimodal Modelscited by 49
  9. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  10. 2024Mitigating Catastrophic Forgetting in Language Transfer via Model Mergingcited by 36
  11. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 23
  12. 2024Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Modelscited by 22
21 more
  1. 2024SMoLoRA: Exploring and Defying Dual Catastrophic Forgetting in Continual Visual Instruction Tuningcited by 19
  2. 2024LLMs Can Evolve Continually on Modality for X-Modal Reasoningcited by 12
  3. 2024Don't Half-listen: Capturing Key-part Information in Continual Instruction Tuningcited by 10
  4. 2023Distilling Rule-based Knowledge into Large Language Modelscited by 9
  5. 2024Adapt-∞: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selectioncited by 9
  6. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 9
  7. 2025TiEBe: Tracking Language Model Recall of Notable Worldwide Events Through Timecited by 8
  8. 2024Towards Incremental Learning in Large Language Models: A Critical Reviewcited by 6
  9. 2026Beyond Static Models: An Evolving Framework for Continual Learning in Large Language Models across Training Stagescited by 6
  10. 2025SEE: Continual Fine-tuning with Sequential Ensemble of Expertscited by 4
  11. 2024Unveiling and Addressing Pseudo Forgetting in Large Language Modelscited by 3
  12. 2024Can LLMs Learn New Concepts Incrementally without Forgetting?cited by 1
  13. 2024HPE-CogVLM: Advancing Vision Language Models with a Head Pose Grounding Taskcited by 1
  14. 2026Continual Learning Mechanisms Compose for Long-Horizon Memorizationcited by 1
  15. 2024Towards Efficient Large Language Models for Scientific Text: A Reviewcited by 0
  16. 2025Enhancing Memory Recall in LLMs with Gauss-Tin: A Hybrid Instructional and Gaussian Replay Approachcited by 0
  17. 2026Continual-NExT: A Unified Comprehension And Generation Continual Learning Frameworkcited by 0
  18. 2026Structural Abstraction as an Inductive Bias for Non-Stationary Language Model Trainingcited by 0
  19. 2026Continual Learning in Transitioncited by 0
  20. 2026Hyper-LLaVA: Hyperbolic Uncertainty-aware Modality-Balanced Routing for Multimodal Continual Instruction Tuningcited by 0
  21. 2026Stabilizing language models under continual learning via condition-anchored distillationcited by 0