mlsanityData to 7 Oct 2026

Temporal Effects on Pre-trained Models for Language Processing Tasks

arXiv 2111.12790 ↗First posted 24 Nov 2021cs.CLcited by 46

Not on the map. Why?

Abstract (arXiv)

Keeping the performance of language technologies optimal as time passes is of great practical interest. We study temporal effects on model performance on downstream language tasks, establishing a nuanced terminology for such discussion and identifying factors essential to conduct a robust study. We present experiments for several tasks in English where the label correctness is not dependent on time and demonstrate the importance of distinguishing between temporal model deterioration and temporal domain adaptation for systems using pre-trained representations. We find that depending on the task, temporal model deterioration is not necessarily a concern. Temporal domain adaptation however is beneficial in all cases, with better performance for a given time period possible when the system is trained on temporally more recent data. Therefore, we also examine the efficacy of two approaches for temporal domain adaptation without human annotations on new data. Self-labeling shows consistent improvement and notably, for named entity recognition, leads to better temporal adaptation than even human annotations.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Builds on 18 of its 56 references that are in the map. First posted on arXiv 24 Nov 2021.

Builds on

18 of 56 references.

References · 18

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  4. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  5. 2018Deep contextualized word representationscited by 2,285
  6. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  7. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  8. 2016Neural Architectures for Named Entity Recognitioncited by 548
  9. 2016End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRFcited by 341
  10. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 290
  11. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
  12. 2016Diachronic Word Embeddings Reveal Statistical Laws of Semantic Changecited by 219
6 more
  1. 2017Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Taggingcited by 71
  2. 2020We Need to Talk About Random Splitscited by 70
  3. 2021Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Mediacited by 51
  4. 2020Dynamic Contextualized Word Embeddingscited by 20
  5. 2020Interpretability Analysis for Named Entity Recognition to Understand System Predictions and How They Can Improvecited by 16
  6. 2021Mitigating Temporal-Drift: A Simple Approach to Keep NER Models Crispcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 12; 2023: 14; 2024: 11; 2025: 5; 2026: 4.

46 papers.

Sort:

By area:

  1. 2022Mass-Editing Memory in a Transformercited by 976
  2. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 180
  3. 2022TimeLMs: Diachronic Language Models from Twittercited by 129
  4. 2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 117
  5. 2023Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluationscited by 96
  6. 2023ALCUNA: Large Language Models Meet New Knowledgecited by 35
  7. 2022Named Entity Recognition in Twitter: A Dataset and Analysis on Short-Term Temporal Shiftscited by 27
  8. 2024Set the Clock: Temporal Alignment of Pretrained Language Modelscited by 24
  9. 2022Temporal Attention for Language Modelscited by 21
  10. 2023Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Viewscited by 21
  11. 2022TempoWiC: An Evaluation Benchmark for Detecting Meaning Shift in Social Mediacited by 19
  12. 2024NEO-BENCH: Evaluating Robustness of Large Language Models with Neologismscited by 19
34 more
  1. 2024Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oraclecited by 19
  2. 2023It's about Time: Rethinking Evaluation on Rumor Detection Benchmarks using Chronological Splitscited by 15
  3. 2023Are Large Language Models Robust Coreference Resolvers?cited by 15
  4. 2022On the Impact of Temporal Concept Drift on Model Explanationscited by 14
  5. 2023TempEL: Linking Dynamically Evolving and Newly Emerging Entitiescited by 14
  6. 2025TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretrainingcited by 13
  7. 2024ChronosLex: Time-aware Incremental Training for Temporal Generalization of Legal Classification Taskscited by 11
  8. 2022Time-aware Prompting for Text Generationcited by 10
  9. 2022Do CoNLL-2003 Named Entity Taggers Still Work Well in 2023?cited by 9
  10. 2023Can LMs Generalize to Future Data? An Empirical Analysis on Text Summarizationcited by 9
  11. 2023SuperTweetEval: A Challenging, Unified and Heterogeneous Benchmark for Social Media NLP Researchcited by 9
  12. 2025Examining and Adapting Time for Multilingual Classification via Mixture of Temporal Expertscited by 9
  13. 2024Time Matters: Examine Temporal Effects on Biomedical Language Modelscited by 7
  14. 2022EDIN: An End-to-end Benchmark and Pipeline for Unknown Entity Discovery and Indexingcited by 5
  15. 2024See the Unseen: Better Context-Consistent Knowledge-Editing by Noisescited by 5
  16. 2025DocMEdit: Towards Document-Level Model Editingcited by 5
  17. 2022Then and Now: Quantifying the Longitudinal Validity of Self-Disclosed Depression Diagnosescited by 4
  18. 2024LLM Surgery: Efficient Knowledge Unlearning and Editing in Large Language Modelscited by 4
  19. 2024FAME: Towards Factual Multi-Task Model Editingcited by 3
  20. 2022Learning Dynamic Contextualised Word Embeddings via Template-based Temporal Adaptationcited by 2
  21. 2023Characterizing and Measuring Linguistic Dataset Driftcited by 1
  22. 2024CHEW: A Dataset of CHanging Events in Wikipediacited by 1
  23. 2026All Leaks Count, Some Count More: Interpretable Temporal Contamination Detection and Mitigation in LLM Backtestingcited by 1
  24. 2026Measuring Distribution Shift in User Prompts and Its Effects on LLM Performancecited by 1
  25. 2026Knowledge-driven Augmentation and Retrieval for Integrative Temporal Adaptationcited by 1
  26. 2023Continuously Reliable Detection of New-Normal Misinformation: Semantic Masking and Contrastive Smoothing in High-Density Latent Regionscited by 0
  27. 2023Boosting classification reliability of NLP transformer models in the long runcited by 0
  28. 2023WIKITIDE: A Wikipedia-Based Timestamped Definition Pairs Datasetcited by 0
  29. 2023Keeping in Time: Adding Temporal Context to Sentiment Analysis Modelscited by 0
  30. 2024Future Language Modeling from Temporal Document Historycited by 0
  31. 2024A Systematic Analysis on the Temporal Generalization of Language Models in Social Mediacited by 0
  32. 2025Market-Derived Financial Sentiment Analysis: Context-Aware Language Models for Crypto Forecastingcited by 0
  33. 2025Decoding the Past: Explainable Machine Learning Models for Dating Historical Textscited by 0
  34. 2026KoNeoBench: A Curated Evaluation Dataset for LLM Understanding of Korean Neologismscited by 0