# Transforming Fake News: Robust Generalisable News Classification Using Transformers

arXiv: https://arxiv.org/abs/2109.09796 · HTML: https://mlsanity.com/paper/2109.09796/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 57 references that are in the map.
- First posted on arXiv 20 Sep 2021.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [Using Pre-Training Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1901.09960/) (2019)
- ["Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection](https://mlsanity.com/paper/1705.00648/) (2017)
- [Deep Learning: A Critical Appraisal](https://mlsanity.com/paper/1801.00631/) (2018)
- [Augmenting Data with Mixup for Sentence Classification: An Empirical Study](https://mlsanity.com/paper/1905.08941/) (2019)
- [Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing Auxiliary Sentence](https://mlsanity.com/paper/1903.09588/) (2019)
- [Beyond News Contents: The Role of Social Context for Fake News Detection](https://mlsanity.com/paper/1712.07709/) (2017)
- [Comparing BERT against traditional machine learning text classification](https://mlsanity.com/paper/2005.13012/) (2020)
- [Rank over Class: The Untapped Potential of Ranking in Natural Language Processing](https://mlsanity.com/paper/2009.05160/) (2020)
- [NIESR: Nuisance Invariant End-to-end Speech Recognition](https://mlsanity.com/paper/1907.03233/) (2019)

## Built on by (most cited first, 8 of 8)

- [A Domain-adaptive Pre-training Approach for Language Bias Detection in News](https://mlsanity.com/paper/2205.10773/) (2022)
- [The Media Bias Taxonomy: A Systematic Literature Review on the Forms and Automated Detection of Media Bias](https://mlsanity.com/paper/2312.16148/) (2023)
- [A Proposed Bi-LSTM Method to Fake News Detection](https://mlsanity.com/paper/2206.13982/) (2022)
- [X-CapsNet For Fake News Detection](https://mlsanity.com/paper/2307.12332/) (2023)
- [An exploration of features to improve the generalisability of fake news detection models](https://mlsanity.com/paper/2502.20299/) (2025)
- [A Unified Training Process for Fake News Detection based on Fine-Tuned BERT Model](https://mlsanity.com/paper/2202.01907/) (2022)
- [Breaking Free Transformer Models: Task-specific Context Attribution Promises Improved Generalizability Without Fine-tuning Pre-trained LLMs](https://mlsanity.com/paper/2401.16638/) (2024)
- [FNDEX: Fake News and Doxxing Detection with Explainable AI](https://mlsanity.com/paper/2410.22390/) (2024)

## Abstract (arXiv)

As online news has become increasingly popular and fake news increasingly prevalent, the ability to audit the veracity of online news content has become more important than ever. Such a task represents a binary classification challenge, for which transformers have achieved state-of-the-art results. Using the publicly available ISOT and Combined Corpus datasets, this study explores transformers' abilities to identify fake news, with particular attention given to investigating generalisation to unseen datasets with varying styles, topics and class distributions. Moreover, we explore the idea that opinion-based news articles cannot be classified as real or fake due to their subjective nature and often sensationalised language, and propose a novel two-step classification pipeline to remove such articles from both model training and the final deployed inference system. Experiments over the ISOT and Combined Corpus datasets show that transformers achieve an increase in F1 scores of up to 4.9% for out of distribution generalisation compared to baseline approaches, with a further increase of 10.1% following the implementation of our two-step classification pipeline. To the best of our knowledge, this study is the first to investigate generalisation of transformers in this context.