# Automatic Fake News Detection: Are Models Learning to Reason?

arXiv: https://arxiv.org/abs/2105.07698 · HTML: https://mlsanity.com/paper/2105.07698/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 26 references that are in the map.
- First posted on arXiv 17 May 2021.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- ["Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection](https://mlsanity.com/paper/1705.00648/) (2017)
- [MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims](https://mlsanity.com/paper/1909.03242/) (2019)
- [Automatic Detection of Fake News](https://mlsanity.com/paper/1708.07104/) (2017)
- [Analyzing the Behavior of Visual Question Answering Models](https://mlsanity.com/paper/1606.07356/) (2016)
- [DeClarE: Debunking Fake News and False Claims using Evidence-Aware Deep Learning](https://mlsanity.com/paper/1809.06416/) (2018)
- [Automated Fact Checking: Task formulations, methods and future directions](https://mlsanity.com/paper/1806.07687/) (2018)
- [That is a Known Lie: Detecting Previously Fact-Checked Claims](https://mlsanity.com/paper/2005.06058/) (2020)
- [Natural Language Inference by Tree-Based Convolution and Heuristic Matching](https://mlsanity.com/paper/1512.08422/) (2015)
- [Integrating Stance Detection and Fact Checking in a Unified Corpus](https://mlsanity.com/paper/1804.08012/) (2018)
- [Lexical Features in Coreference Resolution: To be Used With Caution](https://mlsanity.com/paper/1704.06779/) (2017)
- [Time-Aware Evidence Ranking for Fact-Checking](https://mlsanity.com/paper/2009.06402/) (2020)
- [Neural Check-Worthiness Ranking with Weak Supervision: Finding Sentences for Fact-Checking](https://mlsanity.com/paper/1903.08404/) (2019)
- [DeFactoNLP: Fact Verification using Entity Recognition, TFIDF Vector Comparison and Decomposable Attention](https://mlsanity.com/paper/1809.00509/) (2018)

## Built on by (most cited first, 7 of 7)

- [Missing Counter-Evidence Renders NLP Fact-Checking Unrealistic for Misinformation](https://mlsanity.com/paper/2210.13865/) (2022)
- [Out-of-distribution Evidence-aware Fake News Detection via Dual Adversarial Debiasing](https://mlsanity.com/paper/2304.12888/) (2023)
- [Automatic Fake News Detection: Are current models "fact-checking" or "gut-checking"?](https://mlsanity.com/paper/2204.07229/) (2022)
- [Time Will Change Things: An Empirical Study on Dynamic Language Understanding in Social Media Classification](https://mlsanity.com/paper/2210.02857/) (2022)
- [Tackling Fake News in Bengali: Unraveling the Impact of Summarization vs. Augmentation on Pre-trained Language Models](https://mlsanity.com/paper/2307.06979/) (2023)
- [How Entangled is Factuality and Deception in German?](https://mlsanity.com/paper/2409.20165/) (2024)
- [How Robust Are Automated Fact-Checking Systems? A Cross-Benchmark Evaluation](https://mlsanity.com/paper/2608.25934/) (2026)

## Abstract (arXiv)

Most fact checking models for automatic fake news detection are based on reasoning: given a claim with associated evidence, the models aim to estimate the claim veracity based on the supporting or refuting content within the evidence. When these models perform well, it is generally assumed to be due to the models having learned to reason over the evidence with regards to the claim. In this paper, we investigate this assumption of reasoning, by exploring the relationship and importance of both claim and evidence. Surprisingly, we find on political fact checking datasets that most often the highest effectiveness is obtained by utilizing only the evidence, as the impact of including the claim is either negligible or harmful to the effectiveness. This highlights an important problem in what constitutes evidence in existing approaches for automatic fake news detection.