# The Limitations of Stylometry for Detecting Machine-Generated Fake News

arXiv: https://arxiv.org/abs/1908.09805 · HTML: https://mlsanity.com/paper/1908.09805/

## Key facts

- Cited by 63 papers in the map (data through 13 Oct 2026).
- First posted on arXiv 26 Aug 2019.

## Built on by (most cited first, 30 of 63)

- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- [Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence](https://mlsanity.com/paper/2103.08541/) (2021)
- [Can LLM-Generated Misinformation Be Detected?](https://mlsanity.com/paper/2309.13788/) (2023)
- [FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured information](https://mlsanity.com/paper/2106.05707/) (2021)
- [RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors](https://mlsanity.com/paper/2405.07940/) (2024)
- [TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation](https://mlsanity.com/paper/2109.13296/) (2021)
- [A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions](https://mlsanity.com/paper/2310.14724/) (2023)
- [Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicity](https://mlsanity.com/paper/2301.12867/) (2023)
- [Stylometric Detection of AI-Generated Text in Twitter Timelines](https://mlsanity.com/paper/2303.03697/) (2023)
- [Watermarking Conditional Text Generation for AI Detection: Unveiling Challenges and a Semantic-Aware Watermark Remedy](https://mlsanity.com/paper/2307.13808/) (2023)
- [Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusters](https://mlsanity.com/paper/2204.07447/) (2022)
- [Attribution and Obfuscation of Neural Text Authorship: A Data Mining Perspective](https://mlsanity.com/paper/2210.10488/) (2022)
- [CHEF: A Pilot Chinese Dataset for Evidence-Based Fact-Checking](https://mlsanity.com/paper/2206.11863/) (2022)
- [Misinfo Reaction Frames: Reasoning about Readers' Reactions to News Headlines](https://mlsanity.com/paper/2104.08790/) (2021)
- [CoVERT: A Corpus of Fact-checked Biomedical COVID-19 Tweets](https://mlsanity.com/paper/2204.12164/) (2022)
- [Findings of the The RuATD Shared Task 2022 on Artificial Text Detection in Russian](https://mlsanity.com/paper/2206.01583/) (2022)
- [Embarrassingly Simple Text Watermarks](https://mlsanity.com/paper/2310.08920/) (2023)
- [Domain Adaptive Fake News Detection via Reinforcement Learning](https://mlsanity.com/paper/2202.08159/) (2022)
- [Interpretable Unified Language Checking](https://mlsanity.com/paper/2304.03728/) (2023)
- [How susceptible are LLMs to Logical Fallacies?](https://mlsanity.com/paper/2308.09853/) (2023)
- [Detecting AI-Generated Text: Factors Influencing Detectability with Current Methods](https://mlsanity.com/paper/2406.15583/) (2024)
- [Large Language Models are Advanced Anonymizers](https://mlsanity.com/paper/2402.13846/) (2024)
- [The Intended Uses of Automated Fact-Checking Artefacts: Why, How and Who](https://mlsanity.com/paper/2304.14238/) (2023)
- [SEMQA: Semi-Extractive Multi-Source Question Answering](https://mlsanity.com/paper/2311.04886/) (2023)
- [Fake News Detectors are Biased against Texts Generated by Large Language Models](https://mlsanity.com/paper/2309.08674/) (2023)
- [Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challenges](https://mlsanity.com/paper/2405.15604/) (2024)
- [Read it Twice: Towards Faithfully Interpretable Fact Verification by Revisiting Evidence](https://mlsanity.com/paper/2305.03507/) (2023)
- [COVID-VTS: Fact Extraction and Verification on Short Video Platforms](https://mlsanity.com/paper/2302.07919/) (2023)
- [Knowledge Graph informed Fake News Classification via Heterogeneous Representation Ensembles](https://mlsanity.com/paper/2110.10457/) (2021)
- [Adapting Fake News Detection to the Era of Large Language Models](https://mlsanity.com/paper/2311.04917/) (2023)

## Abstract (arXiv)

Recent developments in neural language models (LMs) have raised concerns about their potential misuse for automatically spreading misinformation. In light of these concerns, several studies have proposed to detect machine-generated fake news by capturing their stylistic differences from human-written text. These approaches, broadly termed stylometry, have found success in source attribution and misinformation detection in human-written texts. However, in this work, we show that stylometry is limited against machine-generated misinformation. While humans speak differently when trying to deceive, LMs generate stylistically consistent text, regardless of underlying motive. Thus, though stylometry can successfully prevent impersonation by identifying text provenance, it fails to distinguish legitimate LM applications from those that introduce false information. We create two benchmarks demonstrating the stylistic similarity between malicious and legitimate uses of LMs, employed in auto-completion and editing-assistance settings. Our findings highlight the need for non-stylometry approaches in detecting machine-generated misinformation, and open up the discussion on the desired evaluation benchmarks.