# Automatic Fake News Detection: Are current models "fact-checking" or "gut-checking"?

arXiv: https://arxiv.org/abs/2204.07229 · HTML: https://mlsanity.com/paper/2204.07229/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 28 references that are in the map.
- First posted on arXiv 14 Apr 2022.

## Builds on (most cited first, 11 of 11)

- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfer](https://mlsanity.com/paper/1804.06437/) (2018)
- [MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims](https://mlsanity.com/paper/1909.03242/) (2019)
- [Automatically Neutralizing Subjective Bias in Text](https://mlsanity.com/paper/1911.09709/) (2019)
- [That is a Known Lie: Detecting Previously Fact-Checked Claims](https://mlsanity.com/paper/2005.06058/) (2020)
- [Fighting the COVID-19 Infodemic: Modeling the Perspective of Journalists, Fact-Checkers, Social Media Platforms, Policy Makers, and the Society](https://mlsanity.com/paper/2005.00033/) (2020)
- [Word Affect Intensities](https://mlsanity.com/paper/1704.08798/) (2017)
- [Where Are the Facts? Searching for Fact-checked Information to Alleviate the Spread of Fake News](https://mlsanity.com/paper/2010.03159/) (2020)
- [Fake News Detection via NLP is Vulnerable to Adversarial Attacks](https://mlsanity.com/paper/1901.09657/) (2019)
- [Automatic Fake News Detection: Are Models Learning to Reason?](https://mlsanity.com/paper/2105.07698/) (2021)
- [DeFactoNLP: Fact Verification using Entity Recognition, TFIDF Vector Comparison and Decomposable Attention](https://mlsanity.com/paper/1809.00509/) (2018)

## Built on by (most cited first, 3 of 3)

- [Emotion Detection for Misinformation: A Review](https://mlsanity.com/paper/2311.00671/) (2023)
- [Capturing Pertinent Symbolic Features for Enhanced Content-Based Misinformation Detection](https://mlsanity.com/paper/2401.16285/) (2024)
- [How Entangled is Factuality and Deception in German?](https://mlsanity.com/paper/2409.20165/) (2024)

## Abstract (arXiv)

Automatic fake news detection models are ostensibly based on logic, where the truth of a claim made in a headline can be determined by supporting or refuting evidence found in a resulting web query. These models are believed to be reasoning in some way; however, it has been shown that these same results, or better, can be achieved without considering the claim at all -- only the evidence. This implies that other signals are contained within the examined evidence, and could be based on manipulable factors such as emotion, sentiment, or part-of-speech (POS) frequencies, which are vulnerable to adversarial inputs. We neutralize some of these signals through multiple forms of both neural and non-neural pre-processing and style transfer, and find that this flattening of extraneous indicators can induce the models to actually require both claims and evidence to perform well. We conclude with the construction of a model using emotion vectors built off a lexicon and passed through an "emotional attention" mechanism to appropriately weight certain emotions. We provide quantifiable results that prove our hypothesis that manipulable features are being used for fact-checking.