mlsanityData to 14 Oct 2026

Carpe Diem: On the Evaluation of World Knowledge in Lifelong Language Models

arXiv 2311.08106 ↗First posted 14 Nov 2023cs.CLcited by 18

Not on the map. Why?

Abstract (arXiv)

The dynamic nature of knowledge in an ever-changing world presents challenges for language models trained on static data; the model in the real world often requires not only acquiring new knowledge but also overwriting outdated information into updated ones. To study the ability of language models for these time-dependent dynamics in human language, we introduce a novel task, EvolvingQA, a temporally evolving question-answering benchmark designed for training and evaluating LMs on an evolving Wikipedia database. The construction of EvolvingQA is automated with our pipeline using large language models. We uncover that existing continual learning baselines suffer from updating and removing outdated knowledge. Our analysis suggests that models fail to rectify knowledge due to small weight gradients. In addition, we elucidate that language models particularly struggle to reflect the change of numerical or temporal information. Our work aims to model the dynamic nature of real-world information, suggesting faithful evaluations of the evolution-adaptability of language models.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 54 of its 62 references that are in the map. First posted on arXiv 14 Nov 2023.

Builds on

54 of 62 references.

References · 54

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  7. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  8. 2016Learning without Forgettingcited by 2,496
  9. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  10. 2019Language Models as Knowledge Bases?cited by 1,936
  11. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  12. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
38 more
  1. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  2. 2016Progressive Neural Networkscited by 1,361
  3. 2017Continual Learning with Deep Generative Replaycited by 1,158
  4. 2018Experience Replay for Continual Learningcited by 1,095
  5. 2022Mass-Editing Memory in a Transformercited by 1,001
  6. 2019How Can We Know What Language Models Know?cited by 959
  7. 2021Learning to Prompt for Continual Learningcited by 944
  8. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 719
  9. 2021Editing Factual Knowledge in Language Modelscited by 553
  10. 2021Fast Model Editing at Scalecited by 498
  11. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  12. 2022Memory-Based Model Editing at Scalecited by 447
  13. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 319
  14. 2023MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questionscited by 300
  15. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 296
  16. 2023Can We Edit Factual Knowledge by In-Context Learning?cited by 285
  17. 2022RealTime QA: What's the Answer Right Now?cited by 244
  18. 2017Overcoming Catastrophic Forgetting by Incremental Moment Matchingcited by 243
  19. 2023Transformer-Patcher: One Mistake worth One Neuroncited by 197
  20. 2023Progressive Prompts: Continual Learning for Language Modelscited by 197
  21. 2019Episodic Memory in Lifelong Language Learningcited by 188
  22. 2020PLOP: Learning without Forgetting for Continual Semantic Segmentationcited by 177
  23. 2021Towards Continual Knowledge Learning of Language Modelscited by 174
  24. 2020Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgettingcited by 170
  25. 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
  26. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 138
  27. 2022StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Modelscited by 126
  28. 2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 119
  29. 2021Online Coreset Selection for Rehearsal-based Continual Learningcited by 117
  30. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 115
  31. 2022ELLE: Efficient Lifelong Pre-training for Emerging Datacited by 70
  32. 2022Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluationcited by 67
  33. 2023Lifelong Language Pretraining with Distribution-Specialized Expertscited by 66
  34. 2021Semantic Answer Similarity for Evaluating Question Answering Modelscited by 47
  35. 2021Adapting BERT for Continual Learning of a Sequence of Aspect Sentiment Classification Taskscited by 46
  36. 2022PIVOT: Prompting for Video Continual Learningcited by 45
  37. 2022On the Soft-Subnetwork for Few-shot Class Incremental Learningcited by 37
  38. 2021Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learningcited by 35

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 12; 2026: 3.

18 papers.

Sort:

By area:

  1. 2025From RAG to Memory: Non-Parametric Continual Learning for Large Language Modelscited by 231
  2. 2024Knowledge Entropy Decay during Language Model Pretraining Hinders New Knowledge Acquisitioncited by 16
  3. 2025Does Time Have Its Place? Temporal Heads: Where Language Models Recall Time-specific Informationcited by 15
  4. 2025TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretrainingcited by 15
  5. 2025HoH: A Dynamic Benchmark for Evaluating the Impact of Outdated Information on Retrieval-Augmented Generationcited by 13
  6. 2024GrowOVER: How Can LLMs Adapt to Growing Real-World Knowledge?cited by 11
  7. 2024ChroKnowledge: Unveiling Chronological Knowledge of Language Models in Multiple Domainscited by 8
  8. 2025Bring Your Own Knowledge: A Survey of Methods for LLM Knowledge Expansioncited by 8
  9. 2025Bidirectional LMs are Better Knowledge Memorizers? A Benchmark for Real-world Knowledge Injectioncited by 7
  10. 2026Can Large Language Models Keep Up? Benchmarking Online Adaptation to Continual Knowledge Streamscited by 5
  11. 2025MegaWika 2: A More Comprehensive Multilingual Collection of Articles and their Sourcescited by 3
  12. 2025When Facts Change: Probing LLMs on Evolving Knowledge with evolveQAcited by 2
6 more
  1. 2026Right Knowledge, Wrong Answer: Characterizing Parametric Temporal Conflict in Open-Weight Language Modelscited by 2
  2. 2025Factual Knowledge in Language Models: Robustness and Anomalies under Simple Temporal Context Variationscited by 1
  3. 2025EMERGE: A Benchmark for Updating Knowledge Graphs with Emerging Textual Knowledgecited by 0
  4. 2025Harnessing Temporal Databases for Systematic Evaluation of Factual Time-Sensitive Question-Answering in Large Language Modelscited by 0
  5. 2025ChronoPlay: A Framework for Modeling Dual Dynamics and Authenticity in Game RAG Benchmarkscited by 0
  6. 2026Fresh at Decision, Stale at Action: Routing New Information to LLM Decisions Already Madecited by 0