# Building for Tomorrow: Assessing the Temporal Persistence of Text Classifiers

arXiv: https://arxiv.org/abs/2205.05435 · HTML: https://mlsanity.com/paper/2205.05435/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 57 references that are in the map.
- First posted on arXiv 11 May 2022.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Mind the Gap: Assessing Temporal Generalization in Neural Language Models](https://mlsanity.com/paper/2102.01951/) (2021)
- [Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change](https://mlsanity.com/paper/1605.09096/) (2016)
- [Advances in Pre-Training Distributed Word Representations](https://mlsanity.com/paper/1712.09405/) (2017)
- [Stance Detection with Bidirectional Conditional Encoding](https://mlsanity.com/paper/1606.05464/) (2016)
- [Diachronic word embeddings and semantic shifts: a survey](https://mlsanity.com/paper/1806.03537/) (2018)
- [Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Media](https://mlsanity.com/paper/2104.08116/) (2021)
- [Time-Aware Evidence Ranking for Fact-Checking](https://mlsanity.com/paper/2009.06402/) (2020)
- [Opinions are Made to be Changed: Temporally Adaptive Stance Classification](https://mlsanity.com/paper/2108.12476/) (2021)
- [Mitigation of Diachronic Bias in Fake News Detection Dataset](https://mlsanity.com/paper/2108.12601/) (2021)
- [Capturing Stance Dynamics in Social Media: Open Challenges and Research Directions](https://mlsanity.com/paper/2109.00475/) (2021)
- [The emojification of sentiment on social media: Collection and analysis of a longitudinal Twitter sentiment dataset](https://mlsanity.com/paper/2108.13898/) (2021)
- [Persistence pays off: Paying Attention to What the LSTM Gating Mechanism Persists](https://mlsanity.com/paper/1810.04437/) (2018)

## Built on by (most cited first, 8 of 8)

- [Large Language Models as Financial Data Annotators: A Study on Effectiveness and Efficiency](https://mlsanity.com/paper/2403.18152/) (2024)
- [Examining Temporal Bias in Abusive Language Detection](https://mlsanity.com/paper/2309.14146/) (2023)
- [Examining Temporalities on Stance Detection towards COVID-19 Vaccination](https://mlsanity.com/paper/2304.04806/) (2023)
- [Cross-lingual Offensive Language Detection: A Systematic Review of Datasets, Transfer Approaches and Challenges](https://mlsanity.com/paper/2401.09244/) (2024)
- [Synergizing Machine Learning & Symbolic Methods: A Survey on Hybrid Approaches to Natural Language Processing](https://mlsanity.com/paper/2401.11972/) (2024)
- [Zero-shot Cross-lingual Stance Detection via Adversarial Language Adaptation](https://mlsanity.com/paper/2404.14339/) (2024)
- [Hatevolution: What Static Benchmarks Don't Tell Us](https://mlsanity.com/paper/2506.12148/) (2025)
- [Decoding the Past: Explainable Machine Learning Models for Dating Historical Texts](https://mlsanity.com/paper/2511.23056/) (2025)

## Abstract (arXiv)

Performance of text classification models tends to drop over time due to changes in data, which limits the lifetime of a pretrained model. Therefore an ability to predict a model's ability to persist over time can help design models that can be effectively used over a longer period of time. In this paper, we provide a thorough discussion into the problem, establish an evaluation setup for the task. We look at this problem from a practical perspective by assessing the ability of a wide range of language models and classification algorithms to persist over time, as well as how dataset characteristics can help predict the temporal stability of different models. We perform longitudinal classification experiments on three datasets spanning between 6 and 19 years, and involving diverse tasks and types of data. By splitting the longitudinal datasets into years, we perform a comprehensive set of experiments by training and testing across data that are different numbers of years apart from each other, both in the past and in the future. This enables a gradual investigation into the impact of the temporal gap between training and test sets on the classification performance, as well as measuring the extent of the persistence over time.