mlsanityData to 7 Oct 2026

Time Waits for No One! Analysis and Challenges of Temporal Misalignment

arXiv 2111.07408 ↗First posted 14 Nov 2021cs.CLcited by 91

Not on the map. Why?

Abstract (arXiv)

When an NLP model is trained on text data from one time period and tested or deployed on data from another, the resulting temporal misalignment can degrade end-task performance. In this work, we establish a suite of eight diverse tasks across different domains (social media, science papers, news, and reviews) and periods of time (spanning five years or more) to quantify the effects of temporal misalignment. Our study is focused on the ubiquitous setting where a pretrained model is optionally adapted through continued domain-specific pretraining, followed by task-specific finetuning. We establish a suite of tasks across multiple domains to study temporal misalignment in modern NLP systems. We find stronger effects of temporal misalignment on task performance than have been previously reported. We also find that, while temporal adaptation through continued pretraining can help, these gains are small compared to task-specific finetuning on data from the target time period. Our findings motivate continued research to improve temporal robustness of NLP models.

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). Builds on 31 of its 64 references that are in the map. First posted on arXiv 14 Nov 2021.

Builds on

31 of 64 references.

References · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2018Deep contextualized word representationscited by 2,285
  4. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  5. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  6. 2020Fact or Fiction: Verifying Scientific Claimscited by 546
  7. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  8. 2020Learning under Concept Drift: A Reviewcited by 470
  9. 2018Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Constructioncited by 321
  10. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 292
  11. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 290
  12. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
19 more
  1. 2016Diachronic Word Embeddings Reveal Statistical Laws of Semantic Changecited by 219
  2. 2019Episodic Memory in Lifelong Language Learningcited by 188
  3. 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
  4. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  5. 2018Construction of the Literature Graph in Semantic Scholarcited by 118
  6. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 114
  7. 2021Detoxifying Language Models Risks Marginalizing Minority Voicescited by 106
  8. 2021Challenges in Automated Debiasing for Toxic Language Detectioncited by 95
  9. 2021Continual Learning for Text Classification with Information Disentanglement Based Regularizationcited by 86
  10. 2018Conversations Gone Awry: Detecting Early Signs of Conversational Failurecited by 73
  11. 2021Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Mediacited by 51
  12. 2018Neural Abstractive Text Summarization with Sequence-to-Sequence Modelscited by 39
  13. 2021Pre-train or Annotate? Domain Adaptation with a Constrained Budgetcited by 27
  14. 2018Making "fetch" happen: The influence of social and linguistic context on nonstandard word growth and declinecited by 15
  15. 2021S2AND: A Benchmark and Evaluation System for Author Name Disambiguationcited by 13
  16. 2019Jointly Extracting and Compressing Documents with Summary State Representationscited by 12
  17. 2021Abolitionist Networks: Modeling Language Change in Nineteenth-Century Activist Newspaperscited by 6
  18. 2016Part-of-Speech Tagging for Historical Englishcited by 3
  19. 2019LeafNATS: An Open-Source Toolkit and Live Demo System for Neural Abstractive Text Summarizationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 14; 2023: 23; 2024: 24; 2025: 15; 2026: 15.

91 papers.

Sort:

By area:

  1. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  2. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
  3. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  4. 2022RealTime QA: What's the Answer Right Now?cited by 239
  5. 2023Query Rewriting for Retrieval-Augmented Large Language Modelscited by 211
  6. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  7. 2022Patching open-vocabulary models by interpolating weightscited by 187
  8. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 180
  9. 2022TimeLMs: Diachronic Language Models from Twittercited by 129
  10. 2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 117
  11. 2023R-Tuning: Instructing Large Language Models to Say `I Don't Know'cited by 106
  12. 2024Tuning Language Models by Proxycited by 88
38 more
  1. 2024Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communitiescited by 84
  2. 2022Continual Learning of Natural Language Processing Tasks: A Surveycited by 83
  3. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 60
  4. 2024Enhancing Contextual Understanding in Large Language Models through Contrastive Decodingcited by 49
  5. 2023RAVEN: In-Context Learning with Retrieval-Augmented Encoder-Decoder Language Modelscited by 36
  6. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
  7. 2023Time is Encoded in the Weights of Finetuned Language Modelscited by 31
  8. 2024Set the Clock: Temporal Alignment of Pretrained Language Modelscited by 24
  9. 2024Knowledge Overshadowing Causes Amalgamated Hallucination in Large Language Modelscited by 24
  10. 2022Exploring The Landscape of Distributional Robustness for Question Answering Modelscited by 23
  11. 2023An Overview on Language Models: Recent Developments and Outlookcited by 22
  12. 2023Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Viewscited by 21
  13. 2023Mitigating Temporal Misalignment by Discarding Outdated Factscited by 21
  14. 2024Who's Who: Large Language Models Meet Knowledge Conflicts in Practicecited by 20
  15. 2022TempoWiC: An Evaluation Benchmark for Detecting Meaning Shift in Social Mediacited by 19
  16. 2024NEO-BENCH: Evaluating Robustness of Large Language Models with Neologismscited by 19
  17. 2026From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanismscited by 19
  18. 2022Improving Temporal Generalization of Pre-trained Language Models with Lexical Semantic Changecited by 18
  19. 2022The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learningcited by 17
  20. 2024Data Authenticity, Consent, & Provenance for AI are all broken: what will it take to fix them?cited by 15
  21. 2022Geographic Adaptation of Pretrained Language Modelscited by 14
  22. 2023TempEL: Linking Dynamically Evolving and Newly Emerging Entitiescited by 14
  23. 2025TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretrainingcited by 13
  24. 2023VIBE: Topic-Driven Temporal Adaptation for Twitter Classificationcited by 12
  25. 2024ChronosLex: Time-aware Incremental Training for Temporal Generalization of Legal Classification Taskscited by 11
  26. 2024GrowOVER: How Can LLMs Adapt to Growing Real-World Knowledge?cited by 11
  27. 2025When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Modelscited by 11
  28. 2023Once Upon a textitTime in textitGraph: Relative-Time Pretraining for Complex Temporal Reasoningcited by 10
  29. 2022Do CoNLL-2003 Named Entity Taggers Still Work Well in 2023?cited by 9
  30. 2023Can LMs Generalize to Future Data? An Empirical Analysis on Text Summarizationcited by 9
  31. 2023SuperTweetEval: A Challenging, Unified and Heterogeneous Benchmark for Social Media NLP Researchcited by 9
  32. 2024On the Robustness of Editing Large Language Modelscited by 9
  33. 2025Bring Your Own Knowledge: A Survey of Methods for LLM Knowledge Expansioncited by 8
  34. 2024NewTerm: Benchmarking Real-Time New Terms for Large Language Models with Annual Updatescited by 7
  35. 2023Online Continual Knowledge Learning for Language Modelscited by 6
  36. 2023Learning Optimal Features via Partial Invariancecited by 5
  37. 2023Evolving Domain Adaptation of Pretrained Language Models for Text Classificationcited by 5
  38. 2025Temporal Alignment of Time Sensitive Facts with Activation Engineeringcited by 4

50 most cited of 91 All 91 (JSON)