# Time Waits for No One! Analysis and Challenges of Temporal Misalignment

arXiv: https://arxiv.org/abs/2111.07408 · HTML: https://mlsanity.com/paper/2111.07408/

## Key facts

- Cited by 91 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 64 references that are in the map.
- First posted on arXiv 14 Nov 2021.

## Builds on (most cited first, 30 of 31)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Fact or Fiction: Verifying Scientific Claims](https://mlsanity.com/paper/2004.14974/) (2020)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [Learning under Concept Drift: A Review](https://mlsanity.com/paper/2004.05785/) (2020)
- [Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction](https://mlsanity.com/paper/1808.09602/) (2018)
- [Time-Aware Language Models as Temporal Knowledge Bases](https://mlsanity.com/paper/2106.15110/) (2021)
- [Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies](https://mlsanity.com/paper/1804.11283/) (2018)
- [Mind the Gap: Assessing Temporal Generalization in Neural Language Models](https://mlsanity.com/paper/2102.01951/) (2021)
- [Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change](https://mlsanity.com/paper/1605.09096/) (2016)
- [Episodic Memory in Lifelong Language Learning](https://mlsanity.com/paper/1906.01076/) (2019)
- [SituatedQA: Incorporating Extra-Linguistic Contexts into QA](https://mlsanity.com/paper/2109.06157/) (2021)
- [DEMix Layers: Disentangling Domains for Modular Language Modeling](https://mlsanity.com/paper/2108.05036/) (2021)
- [Construction of the Literature Graph in Semantic Scholar](https://mlsanity.com/paper/1805.02262/) (2018)
- [Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora](https://mlsanity.com/paper/2110.08534/) (2021)
- [Detoxifying Language Models Risks Marginalizing Minority Voices](https://mlsanity.com/paper/2104.06390/) (2021)
- [Challenges in Automated Debiasing for Toxic Language Detection](https://mlsanity.com/paper/2102.00086/) (2021)
- [Continual Learning for Text Classification with Information Disentanglement Based Regularization](https://mlsanity.com/paper/2104.05489/) (2021)
- [Conversations Gone Awry: Detecting Early Signs of Conversational Failure](https://mlsanity.com/paper/1805.05345/) (2018)
- [Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Media](https://mlsanity.com/paper/2104.08116/) (2021)
- [Neural Abstractive Text Summarization with Sequence-to-Sequence Models](https://mlsanity.com/paper/1812.02303/) (2018)
- [Pre-train or Annotate? Domain Adaptation with a Constrained Budget](https://mlsanity.com/paper/2109.04711/) (2021)
- [Making "fetch" happen: The influence of social and linguistic context on nonstandard word growth and decline](https://mlsanity.com/paper/1802.04140/) (2018)
- [S2AND: A Benchmark and Evaluation System for Author Name Disambiguation](https://mlsanity.com/paper/2103.07534/) (2021)
- [Jointly Extracting and Compressing Documents with Summary State Representations](https://mlsanity.com/paper/1904.02020/) (2019)
- [Abolitionist Networks: Modeling Language Change in Nineteenth-Century Activist Newspapers](https://mlsanity.com/paper/2103.07538/) (2021)
- [Part-of-Speech Tagging for Historical English](https://mlsanity.com/paper/1603.03144/) (2016)

## Built on by (most cited first, 30 of 91)

- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://mlsanity.com/paper/2309.01219/) (2023)
- [Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts](https://mlsanity.com/paper/2305.13300/) (2023)
- [Knowledge Conflicts for LLMs: A Survey](https://mlsanity.com/paper/2403.08319/) (2024)
- [RealTime QA: What's the Answer Right Now?](https://mlsanity.com/paper/2207.13332/) (2022)
- [Query Rewriting for Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2305.14283/) (2023)
- [Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models](https://mlsanity.com/paper/2208.03306/) (2022)
- [Patching open-vocabulary models by interpolating weights](https://mlsanity.com/paper/2208.05592/) (2022)
- [A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity](https://mlsanity.com/paper/2305.13169/) (2023)
- [TimeLMs: Diachronic Language Models from Twitter](https://mlsanity.com/paper/2202.03829/) (2022)
- [TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models](https://mlsanity.com/paper/2204.14211/) (2022)
- [R-Tuning: Instructing Large Language Models to Say `I Don't Know'](https://mlsanity.com/paper/2311.09677/) (2023)
- [Tuning Language Models by Proxy](https://mlsanity.com/paper/2401.08565/) (2024)
- [Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communities](https://mlsanity.com/paper/2407.07791/) (2024)
- [Continual Learning of Natural Language Processing Tasks: A Survey](https://mlsanity.com/paper/2211.12701/) (2022)
- [Towards Lifelong Learning of Large Language Models: A Survey](https://mlsanity.com/paper/2406.06391/) (2024)
- [Enhancing Contextual Understanding in Large Language Models through Contrastive Decoding](https://mlsanity.com/paper/2405.02750/) (2024)
- [RAVEN: In-Context Learning with Retrieval-Augmented Encoder-Decoder Language Models](https://mlsanity.com/paper/2308.07922/) (2023)
- [How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advances](https://mlsanity.com/paper/2310.07343/) (2023)
- [Time is Encoded in the Weights of Finetuned Language Models](https://mlsanity.com/paper/2312.13401/) (2023)
- [Set the Clock: Temporal Alignment of Pretrained Language Models](https://mlsanity.com/paper/2402.16797/) (2024)
- [Knowledge Overshadowing Causes Amalgamated Hallucination in Large Language Models](https://mlsanity.com/paper/2407.08039/) (2024)
- [Exploring The Landscape of Distributional Robustness for Question Answering Models](https://mlsanity.com/paper/2210.12517/) (2022)
- [An Overview on Language Models: Recent Developments and Outlook](https://mlsanity.com/paper/2303.05759/) (2023)
- [Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Views](https://mlsanity.com/paper/2302.12297/) (2023)
- [Mitigating Temporal Misalignment by Discarding Outdated Facts](https://mlsanity.com/paper/2305.14824/) (2023)
- [Who's Who: Large Language Models Meet Knowledge Conflicts in Practice](https://mlsanity.com/paper/2410.15737/) (2024)
- [From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms](https://mlsanity.com/paper/2605.06716/) (2026)
- [NEO-BENCH: Evaluating Robustness of Large Language Models with Neologisms](https://mlsanity.com/paper/2402.12261/) (2024)
- [TempoWiC: An Evaluation Benchmark for Detecting Meaning Shift in Social Media](https://mlsanity.com/paper/2209.07216/) (2022)
- [Improving Temporal Generalization of Pre-trained Language Models with Lexical Semantic Change](https://mlsanity.com/paper/2210.17127/) (2022)

## Abstract (arXiv)

When an NLP model is trained on text data from one time period and tested or deployed on data from another, the resulting temporal misalignment can degrade end-task performance. In this work, we establish a suite of eight diverse tasks across different domains (social media, science papers, news, and reviews) and periods of time (spanning five years or more) to quantify the effects of temporal misalignment. Our study is focused on the ubiquitous setting where a pretrained model is optionally adapted through continued domain-specific pretraining, followed by task-specific finetuning. We establish a suite of tasks across multiple domains to study temporal misalignment in modern NLP systems. We find stronger effects of temporal misalignment on task performance than have been previously reported. We also find that, while temporal adaptation through continued pretraining can help, these gains are small compared to task-specific finetuning on data from the target time period. Our findings motivate continued research to improve temporal robustness of NLP models.