mlsanityData to 7 Oct 2026

TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models

arXiv 2204.14211 ↗First posted 29 Apr 2022cs.CLcited by 117

Not on the map. Why?

Abstract (arXiv)

Language Models (LMs) become outdated as the world changes; they often fail to perform tasks requiring recent factual information which was absent or different during training, a phenomenon called temporal misalignment. This is especially a challenging problem because the research community still lacks a coherent dataset for assessing the adaptability of LMs to frequently-updated knowledge corpus such as Wikipedia. To this end, we introduce TemporalWiki, a lifelong benchmark for ever-evolving LMs that utilizes the difference between consecutive snapshots of English Wikipedia and English Wikidata for training and evaluation, respectively. The benchmark hence allows researchers to periodically track an LM's ability to retain previous knowledge and acquire updated/new knowledge at each point in time. We also find that training an LM on the diff data through continual learning methods achieves similar or better perplexity than on the entire snapshot in our benchmark with 12 times less computational cost, which verifies that factual knowledge in LMs can be safely updated with minimal training data via continual learning. The dataset and the code are available at https://github.com/joeljang/temporalwiki.

Key facts. Cited by 117 papers in the map (data through 7 Oct 2026). Builds on 34 of its 48 references that are in the map. First posted on arXiv 29 Apr 2022.

Builds on

34 of 48 references.

References · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  8. 2019Language Models as Knowledge Bases?cited by 1,922
  9. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  10. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  11. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  12. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
22 more
  1. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  2. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  3. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 349
  4. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 318
  5. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 290
  6. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
  7. 2018A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decaycited by 223
  8. 2021Towards Continual Knowledge Learning of Language Modelscited by 172
  9. 2020Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgettingcited by 169
  10. 2021A Dataset for Answering Time-Sensitive Questionscited by 150
  11. 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
  12. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 137
  13. 2022TimeLMs: Diachronic Language Models from Twittercited by 129
  14. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  15. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 114
  16. 2021Time Waits for No One! Analysis and Challenges of Temporal Misalignmentcited by 91
  17. 2021The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpuscited by 60
  18. 2021Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Mediacited by 51
  19. 2021Time Masking for Temporal Language Modelscited by 46
  20. 2021Temporal Effects on Pre-trained Models for Language Processing Taskscited by 46
  21. 2021FRUIT: Faithfully Reflecting Updated Information in Textcited by 39
  22. 2021Dynamic Language Models for Continuously Evolving Contentcited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 11; 2023: 23; 2024: 41; 2025: 20; 2026: 22.

117 papers.

Sort:

By area:

  1. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  2. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 424
  3. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  4. 2022RealTime QA: What's the Answer Right Now?cited by 239
  5. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  6. 2023Query Rewriting for Retrieval-Augmented Large Language Modelscited by 211
  7. 2022Patching open-vocabulary models by interpolating weightscited by 187
  8. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 180
  9. 2024Continual Learning for Large Language Models: A Surveycited by 144
  10. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 135
  11. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 122
  12. 2024Tug-of-War Between Knowledge: Exploring and Resolving Knowledge Conflicts in Retrieval-Augmented Language Modelscited by 93
38 more
  1. 2022Continual Pre-Training Mitigates Forgetting in Language and Visioncited by 82
  2. 2024Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Modelscited by 69
  3. 2023WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipediacited by 67
  4. 2023Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledgecited by 66
  5. 2023Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinationscited by 63
  6. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 60
  7. 2022Entity Cloze By Date: What LMs Know About Unseen Entitiescited by 55
  8. 2024Enhancing Contextual Understanding in Large Language Models through Contrastive Decodingcited by 49
  9. 2024Blinded by Generated Contexts: How Language Models Merge Generated and Retrieved Contexts When Knowledge Conflicts?cited by 47
  10. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  11. 2022Nonparametric Masked Language Modelingcited by 39
  12. 2023TiC-CLIP: Continual Training of CLIP Modelscited by 38
  13. 2023RAVEN: In-Context Learning with Retrieval-Augmented Encoder-Decoder Language Modelscited by 36
  14. 2023ALCUNA: Large Language Models Meet New Knowledgecited by 35
  15. 2024Online Adaptation of Language Models with a Memory of Amortized Contextscited by 35
  16. 2024MemLLM: Finetuning LLMs to Use An Explicit Read-Write Memorycited by 35
  17. 2023Evaluating the Reversal Curse in Model Editingcited by 34
  18. 2024DYNAMICQA: Tracing Internal Knowledge Conflicts in Language Modelscited by 34
  19. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
  20. 2023Efficient Continual Pre-training for Building Domain Specific Large Language Modelscited by 33
  21. 2023Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Modelscited by 32
  22. 2025WikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMscited by 25
  23. 2024Set the Clock: Temporal Alignment of Pretrained Language Modelscited by 24
  24. 2024DyKnow: Dynamically Verifying Time-Sensitive Factual Knowledge in LLMscited by 23
  25. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 22
  26. 2024Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Modelscited by 22
  27. 2023Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Viewscited by 21
  28. 2024TimeChara: Evaluating Point-in-Time Character Hallucination of Role-Playing Large Language Modelscited by 21
  29. 2025MLLM-CL: Continual Learning for Multimodal Large Language Modelscited by 21
  30. 2023DUnE: Dataset for Unified Editingcited by 19
  31. 2024NEO-BENCH: Evaluating Robustness of Large Language Models with Neologismscited by 19
  32. 2026From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanismscited by 19
  33. 2022NEVIS'22: A Stream of 100 Tasks Sampled from 30 Years of Computer Vision Researchcited by 18
  34. 2023Recyclable Tuning for Continual Pre-trainingcited by 17
  35. 2023Carpe Diem: On the Evaluation of World Knowledge in Lifelong Language Modelscited by 17
  36. 2026Pretrained Vision-Language-Action Models are Surprisingly Resistant to Forgetting in Continual Learningcited by 15
  37. 2023TempEL: Linking Dynamically Evolving and Newly Emerging Entitiescited by 14
  38. 2024VersiCode: Towards Version-controllable Code Generationcited by 14

50 most cited of 117 All 117 (JSON)