# The Surprising Performance of Simple Baselines for Misinformation Detection

arXiv: https://arxiv.org/abs/2104.06952 · HTML: https://mlsanity.com/paper/2104.06952/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 106 references that are in the map.
- First posted on arXiv 14 Apr 2021.

## Builds on (most cited first, 30 of 50)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [AllenNLP: A Deep Semantic Natural Language Processing Platform](https://mlsanity.com/paper/1803.07640/) (2018)
- [BERTweet: A pre-trained language model for English Tweets](https://mlsanity.com/paper/2005.10200/) (2020)
- [Fake News Detection on Social Media: A Data Mining Perspective](https://mlsanity.com/paper/1708.01967/) (2017)
- [DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations](https://mlsanity.com/paper/2006.03659/) (2020)
- [Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing](https://mlsanity.com/paper/2006.03236/) (2020)
- [Deep Learning Based Text Classification: A Comprehensive Review](https://mlsanity.com/paper/2004.03705/) (2020)
- [GCAN: Graph-aware Co-Attention Networks for Explainable Fake News Detection on Social Media](https://mlsanity.com/paper/2004.11648/) (2020)
- [COVID-Twitter-BERT: A Natural Language Processing Model to Analyse COVID-19 Content on Twitter](https://mlsanity.com/paper/2005.07503/) (2020)
- [CoAID: COVID-19 Healthcare Misinformation Dataset](https://mlsanity.com/paper/2006.00885/) (2020)
- [Analysing How People Orient to and Spread Rumours in Social Media by Looking at Conversational Threads](https://mlsanity.com/paper/1511.07487/) (2015)
- [The COVID-19 Social Media Infodemic](https://mlsanity.com/paper/2003.05004/) (2020)
- [All-in-one: Multi-task Learning for Rumour Verification](https://mlsanity.com/paper/1806.03713/) (2018)
- [Twitter as a Lifeline: Human-annotated Twitter Corpora for NLP of Crisis-related Messages](https://mlsanity.com/paper/1605.05894/) (2016)
- [Graph Neural Networks with Continual Learning for Fake News Detection from Social Media](https://mlsanity.com/paper/2007.03316/) (2020)
- [A Survey of Fake News: Fundamental Theories, Detection Methods, and Opportunities](https://mlsanity.com/paper/1812.00315/) (2018)
- [Which *BERT? A Survey Organizing Contextualized Encoders](https://mlsanity.com/paper/2010.00854/) (2020)

## Built on by (most cited first, 28 of 28)

- [Fake News in Sheep's Clothing: Robust Fake News Detection Against LLM-Empowered Style Attacks](https://mlsanity.com/paper/2310.10830/) (2023)
- [garak: A Framework for Security Probing Large Language Models](https://mlsanity.com/paper/2406.11036/) (2024)
- [Towards Reliable Misinformation Mitigation: Generalization, Uncertainty, and GPT-4](https://mlsanity.com/paper/2305.14928/) (2023)
- [DELL: Generating Reactions and Explanations for LLM-Based Misinformation Detection](https://mlsanity.com/paper/2402.10426/) (2024)
- [Open, Closed, or Small Language Models for Text Classification?](https://mlsanity.com/paper/2308.10092/) (2023)
- [Towards Fine-Grained Reasoning for Fake News Detection](https://mlsanity.com/paper/2110.15064/) (2021)
- [Combining Confidence Elicitation and Sample-based Methods for Uncertainty Quantification in Misinformation Mitigation](https://mlsanity.com/paper/2401.08694/) (2024)
- [Prompt-and-Align: Prompt-Based Social Alignment for Few-Shot Fake News Detection](https://mlsanity.com/paper/2309.16424/) (2023)
- [A Guide to Misinformation Detection Data and Evaluation](https://mlsanity.com/paper/2411.05060/) (2024)
- [Heterogeneous Subgraph Transformer for Fake News Detection](https://mlsanity.com/paper/2404.13192/) (2024)
- [ContCommRTD: A Distributed Content-based Misinformation-aware Community Detection System for Real-Time Disaster Reporting](https://mlsanity.com/paper/2301.12984/) (2023)
- [Robust Fake News Detection using Large Language Models under Adversarial Sentiment Attacks](https://mlsanity.com/paper/2601.15277/) (2026)
- [Probing Spurious Correlations in Popular Event-Based Rumor Detection Benchmarks](https://mlsanity.com/paper/2209.08799/) (2022)
- [Harnessing the Power of Text-image Contrastive Models for Automatic Detection of Online Misinformation](https://mlsanity.com/paper/2304.10249/) (2023)
- [Global-Liar: Factuality of LLMs over Time and Geographic Regions](https://mlsanity.com/paper/2401.17839/) (2024)
- [RAMA: Retrieval-Augmented Multi-Agent Framework for Misinformation Detection in Multimodal Fact-Checking](https://mlsanity.com/paper/2507.09174/) (2025)
- [Mining User-aware Multi-relations for Fake News Detection in Large Scale Online Social Networks](https://mlsanity.com/paper/2212.10778/) (2022)
- [Detecting misinformation through Framing Theory: the Frame Element-based Model](https://mlsanity.com/paper/2402.15525/) (2024)
- [Evaluating BERT-based Pre-training Language Models for Detecting Misinformation](https://mlsanity.com/paper/2203.07731/) (2022)
- [Model Attribution in LLM-Generated Disinformation: A Domain Generalization Approach with Supervised Contrastive Learning](https://mlsanity.com/paper/2407.21264/) (2024)
- [FNDaaS: Content-agnostic Detection of Fake News sites](https://mlsanity.com/paper/2212.06492/) (2022)
- [DDNAS: Discretized Differentiable Neural Architecture Search for Text Classification](https://mlsanity.com/paper/2307.06005/) (2023)
- [TEGRA: Text Encoding With Graph and Retrieval Augmentation for Misinformation Detection](https://mlsanity.com/paper/2602.11106/) (2026)
- [Reliable News or Propagandist News? A Neurosymbolic Model Using Genre, Topic, and Persuasion Techniques to Improve Robustness in Classification](https://mlsanity.com/paper/2604.01936/) (2026)
- [Predicting Viral Rumors and Vulnerable Users for Infodemic Surveillance](https://mlsanity.com/paper/2401.09724/) (2024)
- [Rumour Evaluation with Very Large Language Models](https://mlsanity.com/paper/2404.16859/) (2024)
- [Understanding Inequality of LLM Fact-Checking over Geographic Regions with Agent and Retrieval models](https://mlsanity.com/paper/2503.22877/) (2025)
- [Wisdom of the LLM Crowd: A Large Scale Benchmark of Multi-Label U.S. Election-Related Harmful Social Media Content](https://mlsanity.com/paper/2602.11962/) (2026)

## Abstract (arXiv)

As social media becomes increasingly prominent in our day to day lives, it is increasingly important to detect informative content and prevent the spread of disinformation and unverified rumours. While many sophisticated and successful models have been proposed in the literature, they are often compared with older NLP baselines such as SVMs, CNNs, and LSTMs. In this paper, we examine the performance of a broad set of modern transformer-based language models and show that with basic fine-tuning, these models are competitive with and can even significantly outperform recently proposed state-of-the-art methods. We present our framework as a baseline for creating and evaluating new methods for misinformation detection. We further study a comprehensive set of benchmark datasets, and discuss potential data leakage and the need for careful design of the experiments and understanding of datasets to account for confounding variables. As an extreme case example, we show that classifying only based on the first three digits of tweet ids, which contain information on the date, gives state-of-the-art performance on a commonly used benchmark dataset for fake news detection --Twitter16. We provide a simple tool to detect this problem and suggest steps to mitigate it in future datasets.