mlsanityData to 14 Oct 2026

Mind the Gap: Assessing Temporal Generalization in Neural Language Models

arXiv 2102.01951 ↗First posted 3 Feb 2021cs.CL cs.AIcited by 230

Not on the map. Why?

Abstract (arXiv)

Our world is open-ended, non-stationary, and constantly evolving; thus what we talk about and how we talk about it change over time. This inherent dynamic nature of language contrasts with the current static language modelling paradigm, which trains and evaluates models on utterances from overlapping time periods. Despite impressive recent progress, we demonstrate that Transformer-XL language models perform worse in the realistic setup of predicting future utterances from beyond their training period, and that model performance becomes increasingly worse with time. We find that, while increasing model size alone -- a key driver behind recent progress -- does not solve this problem, having models that continually update their knowledge with new information can indeed mitigate this performance degradation over time. Hence, given the compilation of ever-larger language modelling datasets, combined with the growing list of language-model-based NLP applications that require up-to-date factual knowledge about the world, we argue that now is the right time to rethink the static way in which we currently train and evaluate our language models, and develop adaptive language models that can remain up-to-date with respect to our ever-changing and non-stationary world. We publicly release our dynamic, streaming language modelling benchmarks for WMT and arXiv to facilitate language model evaluation that takes temporal dynamics into account.

Key facts. Cited by 230 papers in the map (data through 14 Oct 2026). Builds on 53 of its 105 references that are in the map. First posted on arXiv 3 Feb 2021.

Builds on

53 of 105 references.

References · 53

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  9. 2020Longformer: The Long-Document Transformercited by 3,413
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
38 more
  1. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  3. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  6. 2020Reformer: The Efficient Transformercited by 1,668
  7. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  8. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  9. 2016Progressive Neural Networkscited by 1,361
  10. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  11. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  12. 2015Pointer Networkscited by 895
  13. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  14. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  15. 2019Defending Against Neural Fake Newscited by 559
  16. 2013Generating Sequences With Recurrent Neural Networkscited by 554
  17. 2020Learning under Concept Drift: A Reviewcited by 477
  18. 2016NewsQA: A Machine Comprehension Datasetcited by 331
  19. 2018Video Action Transformer Networkcited by 316
  20. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 308
  21. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 294
  22. 2017SearchQA: A New Q&A Dataset Augmented with Context from a Search Enginecited by 225
  23. 2020Modifying Memories in Transformer Modelscited by 224
  24. 2016Diachronic Word Embeddings Reveal Statistical Laws of Semantic Changecited by 221
  25. 2019Episodic Memory in Lifelong Language Learningcited by 188
  26. 2019MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claimscited by 169
  27. 2019LAMOL: LAnguage MOdeling for Lifelong Language Learningcited by 160
  28. 2019Adaptively Sparse Transformerscited by 155
  29. 2020Editable Neural Networkscited by 155
  30. 2019Distributionally Robust Language Modelingcited by 143
  31. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 138
  32. 2017Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environmentscited by 108
  33. 2018Automated Fact Checking: Task formulations, methods and future directionscited by 96
  34. 2020We Need to Talk About Random Splitscited by 71
  35. 2020DynaSent: A Dynamic Benchmark for Sentiment Analysiscited by 63
  36. 2017Dynamic Evaluation of Neural Sequence Modelscited by 55
  37. 2019What Kind of Language Is Hard to Language-Model?cited by 50
  38. 2019Dynamic Evaluation of Transformer Language Modelscited by 23

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 10; 2022: 42; 2023: 53; 2024: 49; 2025: 31; 2026: 45.

230 papers.

Sort:

By area:

  1. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  2. 2022Mass-Editing Memory in a Transformercited by 1,001
  3. 2021Fast Model Editing at Scalecited by 498
  4. 2022Memory-Based Model Editing at Scalecited by 447
  5. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 437
  6. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
  7. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 341
  8. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 296
  9. 2024Knowledge Conflicts for LLMs: A Surveycited by 283
  10. 2022Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptorscited by 271
  11. 2022RealTime QA: What's the Answer Right Now?cited by 244
  12. 2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 242
38 more
  1. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
  2. 2021Internet-Augmented Dialogue Generationcited by 217
  3. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 191
  4. 2022Patching open-vocabulary models by interpolating weightscited by 190
  5. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 182
  6. 2022Large Language Models with Controllable Working Memorycited by 169
  7. 2023Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attackscited by 158
  8. 2022A Review on Language Models as Knowledge Basescited by 157
  9. 2022StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Modelscited by 126
  10. 2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 119
  11. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
  12. 2021Time Waits for No One! Analysis and Challenges of Temporal Misalignmentcited by 91
  13. 2022Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Timecited by 90
  14. 2024Tuning Language Models by Proxycited by 89
  15. 2022Continual Pre-Training Mitigates Forgetting in Language and Visioncited by 83
  16. 2023Context-faithful Prompting for Large Language Modelscited by 80
  17. 2021Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefscited by 79
  18. 2024Towards Conversational Diagnostic AIcited by 79
  19. 2023Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledgecited by 67
  20. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 64
  21. 2023Continual Learning: Applications and the Road Forwardcited by 59
  22. 2024Investigating Continual Pretraining in Large Language Models: Insights and Implicationscited by 57
  23. 2022Entity Cloze By Date: What LMs Know About Unseen Entitiescited by 55
  24. 2025SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviorscited by 52
  25. 2023Faithful Persona-based Conversational Dataset Generation with Large Language Modelscited by 47
  26. 2021Temporal Effects on Pre-trained Models for Language Processing Taskscited by 46
  27. 2023Meta-Learning Online Adaptation of Language Modelscited by 43
  28. 2023Compressed Context Memory For Online Language Model Interactioncited by 43
  29. 2023Diagnosing Model Performance Under Distribution Shiftcited by 39
  30. 2024AdaCAD: Adaptively Decoding to Balance Conflicts between Contextual and Parametric Knowledgecited by 37
  31. 2025A Reasoning-Focused Legal Retrieval Benchmarkcited by 37
  32. 2024Stable Neural Stochastic Differential Equations in Analyzing Irregular Time Series Datacited by 36
  33. 2023ALCUNA: Large Language Models Meet New Knowledgecited by 35
  34. 2024Online Adaptation of Language Models with a Memory of Amortized Contextscited by 35
  35. 2024Benchmarks as Microscopes: A Call for Model Metrologycited by 35
  36. 2023A Survey on Knowledge Editing of Neural Networkscited by 34
  37. 2025Reinforced Lifelong Editing for Language Modelscited by 34
  38. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33

50 most cited of 230 All 230 (JSON)