mlsanityData to 7 Oct 2026

Time-Aware Language Models as Temporal Knowledge Bases

arXiv 2106.15110 ↗First posted 29 Jun 2021cs.CLcited by 290

Not on the map. Why?

Cited by 28 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Many facts come with an expiration date, from the name of the President to the basketball team Lebron James plays for. But language models (LMs) are trained on snapshots of data collected at a specific moment in time, and this can limit their utility, especially in the closed-book setting where the pretraining corpus must contain the facts the model should memorize. We introduce a diagnostic dataset aimed at probing LMs for factual knowledge that changes over time and highlight problems with LMs at either end of the spectrum -- those trained on specific slices of temporal data, as well as those trained on a wide range of temporal data. To mitigate these problems, we propose a simple technique for jointly modeling text with its timestamp. This improves memorization of seen facts from the training time period, as well as calibration on predictions about unseen facts from future time periods. We also show that models trained with temporal context can be efficiently "refreshed" as new data arrives, without the need for retraining from scratch.

Key facts. Cited by 290 papers in the map (data through 7 Oct 2026). Builds on 37 of its 55 references that are in the map. First posted on arXiv 29 Jun 2021.

Builds on

37 of 55 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  7. 2018Deep contextualized word representationscited by 2,285
  8. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  9. 2019Language Models as Knowledge Bases?cited by 1,922
  10. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  11. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  12. 2019How Can We Know What Language Models Know?cited by 953
25 more
  1. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  2. 2021Editing Factual Knowledge in Language Modelscited by 548
  3. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  4. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
  5. 2020Modifying Memories in Transformer Modelscited by 222
  6. 2016Diachronic Word Embeddings Reveal Statistical Laws of Semantic Changecited by 219
  7. 2020Editable Neural Networkscited by 154
  8. 2021Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Modelscited by 139
  9. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 137
  10. 2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
  11. 2019"Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understandingcited by 121
  12. 2020X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Modelscited by 117
  13. 2021Question Answering Over Temporal Knowledge Graphscited by 99
  14. 2020Analysing Lexical Semantic Change with Contextualised Word Representationscited by 92
  15. 2020TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questionscited by 65
  16. 2020Temporal Reasoning on Implicit Events from Distant Supervisioncited by 58
  17. 2017Dynamic Evaluation of Neural Sequence Modelscited by 53
  18. 2017Dynamic Word Embeddingscited by 51
  19. 2020Language Models as Fact Checkers?cited by 51
  20. 2021Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Mediacited by 51
  21. 2019Time-Out: Temporal Referencing for Robust Modeling of Lexical Semantic Changecited by 32
  22. 2021Dynamic Language Models for Continuously Evolving Contentcited by 27
  23. 2017SLING: A framework for frame semantic parsingcited by 12
  24. 2021NewsEmbed: Modeling News through Pre-trained Document Representationscited by 6
  25. 2019Learning Dynamic Author Representations with Temporal Language Modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 41; 2023: 57; 2024: 76; 2025: 55; 2026: 50.

290 papers.

Sort:

By area:

  1. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  2. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  3. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  4. 2023Can We Edit Factual Knowledge by In-Context Learning?cited by 284
  5. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  6. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 273
  7. 2022RealTime QA: What's the Answer Right Now?cited by 239
  8. 2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 238
  9. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  10. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 180
  11. 2023Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Modelscited by 180
  12. 2021Towards Continual Knowledge Learning of Language Modelscited by 172
38 more
  1. 2022Large Language Models with Controllable Working Memorycited by 169
  2. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  3. 2022A Review on Language Models as Knowledge Basescited by 156
  4. 2021A Dataset for Answering Time-Sensitive Questionscited by 150
  5. 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
  6. 2022TimeLMs: Diachronic Language Models from Twittercited by 129
  7. 2022StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Modelscited by 123
  8. 2025BIG-Bench Extra Hardcited by 123
  9. 2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 117
  10. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 114
  11. 2022DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answeringcited by 93
  12. 2021Time Waits for No One! Analysis and Challenges of Temporal Misalignmentcited by 91
  13. 2023KoLA: Carefully Benchmarking World Knowledge of Large Language Modelscited by 85
  14. 2024Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communitiescited by 84
  15. 2022Mind's Eye: Grounded Language Model Reasoning through Simulationcited by 83
  16. 2022Continual Learning of Natural Language Processing Tasks: A Surveycited by 83
  17. 2024Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoningcited by 83
  18. 2022Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Surveycited by 80
  19. 2021Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefscited by 79
  20. 2023Graph-ToolFormer: To Empower LLMs with Graph Reasoning Ability via Prompt Augmented by ChatGPTcited by 78
  21. 2024Simple Is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generationcited by 78
  22. 2022ELLE: Efficient Lifelong Pre-training for Emerging Datacited by 70
  23. 2021Complex Temporal Question Answering on Knowledge Graphscited by 69
  24. 2023Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledgecited by 66
  25. 2023RET-LLM: Towards a General Read-Write Memory for Large Language Modelscited by 64
  26. 2024MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learningcited by 63
  27. 2023GeoLLM: Extracting Geospatial Knowledge from Large Language Modelscited by 62
  28. 2022Entity Cloze By Date: What LMs Know About Unseen Entitiescited by 55
  29. 2023Test of Time: Instilling Video-Language Models with a Sense of Timecited by 51
  30. 2021Time Masking for Temporal Language Modelscited by 46
  31. 2021Temporal Effects on Pre-trained Models for Language Processing Taskscited by 46
  32. 2023Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Modelscited by 46
  33. 2024RA-ISF: Learning to Answer and Understand from Retrieval Augmentation via Iterative Self-Feedbackcited by 46
  34. 2023Meta-Learning Online Adaptation of Language Modelscited by 43
  35. 2021FRUIT: Faithfully Reflecting Updated Information in Textcited by 39
  36. 2022Nonparametric Masked Language Modelingcited by 39
  37. 2023TRAM: Benchmarking Temporal Reasoning for Large Language Modelscited by 39
  38. 2023Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Modelscited by 37

50 most cited of 290 All 290 (JSON)