mlsanityData to 14 Oct 2026

Online Continual Knowledge Learning for Language Models

arXiv 2311.09632 ↗First posted 16 Nov 2023cs.CL cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) serve as repositories of extensive world knowledge, enabling them to perform tasks such as question-answering and fact-checking. However, this knowledge can become obsolete as global contexts change. In this paper, we introduce a novel problem in the realm of continual learning: Online Continual Knowledge Learning (OCKL). This problem formulation aims to manage the dynamic nature of world knowledge in LMs under real-time constraints. We propose a new benchmark and evaluation metric designed to measure both the rate of new knowledge acquisition and the retention of previously learned knowledge. Our empirical evaluation, conducted using a variety of state-of-the-art methods, establishes robust base-lines for OCKL. Our results reveal that existing continual learning approaches are unfortunately insufficient for tackling the unique challenges posed by OCKL. We identify key factors that influence the trade-off between knowledge acquisition and retention, thereby advancing our understanding of how to train LMs in a continually evolving environment.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 41 of its 56 references that are in the map. First posted on arXiv 16 Nov 2023.

Builds on

41 of 56 references.

References · 41

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  7. 2016Learning without Forgettingcited by 2,496
  8. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  9. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  10. 2019Language Models as Knowledge Bases?cited by 1,936
  11. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  12. 2020Knowledge Distillation: A Surveycited by 1,603
29 more
  1. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  2. 2016Progressive Neural Networkscited by 1,361
  3. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  4. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 911
  5. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
  6. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  7. 2019Gradient based sample selection for online continual learningcited by 571
  8. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  9. 2021Editing Factual Knowledge in Language Modelscited by 553
  10. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 356
  11. 2019Learning Loss for Active Learningcited by 336
  12. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 328
  13. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 319
  14. 2019On Tiny Episodic Memories in Continual Learningcited by 307
  15. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 296
  16. 2020Modifying Memories in Transformer Modelscited by 224
  17. 2021New Insights on Reducing Abrupt Representation Change in Online Continual Learningcited by 219
  18. 2020Coresets via Bilevel Optimization for Continual Learning and Streamingcited by 192
  19. 2021Towards Continual Knowledge Learning of Language Modelscited by 174
  20. 2020Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgettingcited by 170
  21. 2020Using Hindsight to Anchor Past Knowledge in Continual Learningcited by 161
  22. 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
  23. 2021Online Coreset Selection for Rehearsal-based Continual Learningcited by 117
  24. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 115
  25. 2021GCR: Gradient Coreset Based Replay Buffer Selection For Continual Learningcited by 95
  26. 2021Time Waits for No One! Analysis and Challenges of Temporal Misalignmentcited by 91
  27. 2023Real-Time Evaluation in Online Continual Learning: A New Hopecited by 47
  28. 2020Experience Replay with Likelihood-free Importance Weightscited by 42
  29. 2023Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Viewscited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 1; 2026: 2.

6 papers.

  1. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  2. 2024Knowledge Entropy Decay during Language Model Pretraining Hinders New Knowledge Acquisitioncited by 16
  3. 2026Can Large Language Models Keep Up? Benchmarking Online Adaptation to Continual Knowledge Streamscited by 5
  4. 2024Unlearning Climate Misinformation in Large Language Modelscited by 4
  5. 2025CurLL: A Developmental Framework to Evaluate Continual Learning in Language Modelscited by 1
  6. 2026Structural Abstraction as an Inductive Bias for Non-Stationary Language Model Trainingcited by 0