# It's about Time: Rethinking Evaluation on Rumor Detection Benchmarks using Chronological Splits

arXiv: https://arxiv.org/abs/2302.03147 · HTML: https://mlsanity.com/paper/2302.03147/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 49 references that are in the map.
- First posted on arXiv 6 Feb 2023.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BERTweet: A pre-trained language model for English Tweets](https://mlsanity.com/paper/2005.10200/) (2020)
- [ERNIE 2.0: A Continual Pre-training Framework for Language Understanding](https://mlsanity.com/paper/1907.12412/) (2019)
- [Rumor Detection on Social Media with Bi-Directional Graph Convolutional Networks](https://mlsanity.com/paper/2001.06362/) (2020)
- [GCAN: Graph-aware Co-Attention Networks for Explainable Fake News Detection on Social Media](https://mlsanity.com/paper/2004.11648/) (2020)
- [Detection and Resolution of Rumours in Social Media: A Survey](https://mlsanity.com/paper/1704.00656/) (2017)
- [We Need to Talk About Random Splits](https://mlsanity.com/paper/2005.00636/) (2020)
- [Analysing How People Orient to and Spread Rumours in Social Media by Looking at Conversational Threads](https://mlsanity.com/paper/1511.07487/) (2015)
- [Temporal Effects on Pre-trained Models for Language Processing Tasks](https://mlsanity.com/paper/2111.12790/) (2021)
- [Towards Propagation Uncertainty: Edge-enhanced Bayesian Graph Convolutional Networks for Rumor Detection](https://mlsanity.com/paper/2107.11934/) (2021)
- [Zoom Out and Observe: News Environment Perception for Fake News Detection](https://mlsanity.com/paper/2203.10885/) (2022)
- [Rumor Detection on Twitter with Claim-Guided Hierarchical Graph Attention Networks](https://mlsanity.com/paper/2110.04522/) (2021)
- [Improved Multi-label Classification under Temporal Concept Drift: Rethinking Group-Robust Algorithms in a Label-Wise Setting](https://mlsanity.com/paper/2203.07856/) (2022)
- [On the Impact of Temporal Concept Drift on Model Explanations](https://mlsanity.com/paper/2210.09197/) (2022)
- [Meet The Truth: Leverage Objective Facts and Subjective Views for Interpretable Rumor Detection](https://mlsanity.com/paper/2107.10747/) (2021)
- [Estimating predictive uncertainty for rumour verification models](https://mlsanity.com/paper/2005.07174/) (2020)
- [Identifying and Characterizing Active Citizens who Refute Misinformation in Social Media](https://mlsanity.com/paper/2204.10080/) (2022)
- [Identifying and Understanding User Reactions to Deceptive and Trusted Social News Sources](https://mlsanity.com/paper/1805.12032/) (2018)
- [MTLTS: A Multi-Task Framework To Obtain Trustworthy Summaries From Crisis-Related Microblogs](https://mlsanity.com/paper/2112.05798/) (2021)

## Built on by (most cited first, 15 of 15)

- [Bad Actor, Good Advisor: Exploring the Role of Large Language Models in Fake News Detection](https://mlsanity.com/paper/2309.12247/) (2023)
- [The Stepwise Deception: Simulating the Evolution from True News to Fake News with LLM Agents](https://mlsanity.com/paper/2410.19064/) (2024)
- [Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Views](https://mlsanity.com/paper/2302.12297/) (2023)
- [Learn over Past, Evolve for Future: Forecasting Temporal Trends for Fake News Detection](https://mlsanity.com/paper/2306.14728/) (2023)
- [Combating Online Misinformation Videos: Characterization, Detection, and Future Directions](https://mlsanity.com/paper/2302.03242/) (2023)
- [Exploiting User Comments for Early Detection of Fake News Prior to Users' Commenting](https://mlsanity.com/paper/2310.10429/) (2023)
- [ChronosLex: Time-aware Incremental Training for Temporal Generalization of Legal Classification Tasks](https://mlsanity.com/paper/2405.14211/) (2024)
- [EUROPA: A Legal Multilingual Keyphrase Generation Dataset](https://mlsanity.com/paper/2403.00252/) (2024)
- [Fine-Tuned LLMs are "Time Capsules" for Tracking Societal Bias Through Books](https://mlsanity.com/paper/2502.05331/) (2025)
- [Examining Temporal Bias in Abusive Language Detection](https://mlsanity.com/paper/2309.14146/) (2023)
- [Examining Temporalities on Stance Detection towards COVID-19 Vaccination](https://mlsanity.com/paper/2304.04806/) (2023)
- [Improving Multimodal Classification of Social Media Posts by Leveraging Image-Text Auxiliary Tasks](https://mlsanity.com/paper/2309.07794/) (2023)
- [Examining the Limitations of Computational Rumor Detection Models Trained on Static Datasets](https://mlsanity.com/paper/2309.11576/) (2023)
- [Enhancing Data Quality through Simple De-duplication: Navigating Responsible Computational Social Science Research](https://mlsanity.com/paper/2410.03545/) (2024)
- [Framework of Voting Prediction of Parliament Members](https://mlsanity.com/paper/2505.12535/) (2025)

## Abstract (arXiv)

New events emerge over time influencing the topics of rumors in social media. Current rumor detection benchmarks use random splits as training, development and test sets which typically results in topical overlaps. Consequently, models trained on random splits may not perform well on rumor classification on previously unseen topics due to the temporal concept drift. In this paper, we provide a re-evaluation of classification models on four popular rumor detection benchmarks considering chronological instead of random splits. Our experimental results show that the use of random splits can significantly overestimate predictive performance across all datasets and models. Therefore, we suggest that rumor detection models should always be evaluated using chronological splits for minimizing topical overlaps.