# Do small language models generate realistic variable-quality fake news headlines?

arXiv: https://arxiv.org/abs/2509.00680 · HTML: https://mlsanity.com/paper/2509.00680/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 9 references that are in the map.
- First posted on arXiv 31 Aug 2025.

## Builds on (most cited first, 7 of 7)

- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Phi-4 Technical Report](https://mlsanity.com/paper/2412.08905/) (2024)
- [Small Language Models: Survey, Measurements, and Insights](https://mlsanity.com/paper/2409.15790/) (2024)
- [Large language models can consistently generate high-quality content for election disinformation operations](https://mlsanity.com/paper/2408.06731/) (2024)
- [Comparing the willingness to share for human-generated vs. AI-generated fake news](https://mlsanity.com/paper/2402.07395/) (2024)
- [Did ChatGPT or Copilot use alter the style of internet news headlines? A time series regression analysis](https://mlsanity.com/paper/2503.23811/) (2025)
- [Binary classification for perceived quality of headlines and links on worldwide news websites, 2018-2024](https://mlsanity.com/paper/2506.09381/) (2025)

## Abstract (arXiv)

Small language models (SLMs) have the capability for text generation and may potentially be used to generate falsified texts online. This study evaluates 14 SLMs (1.7B-14B parameters) including LLaMA, Gemma, Phi, SmolLM, Mistral, and Granite families in generating perceived low and high quality fake news headlines when explicitly prompted, and whether they appear to be similar to real-world news headlines. Using controlled prompt engineering, 24,000 headlines were generated across low-quality and high-quality deceptive categories. Existing machine learning and deep learning-based news headline quality detectors were then applied against these SLM-generated fake news headlines. SLMs demonstrated high compliance rates with minimal ethical resistance, though there were some occasional exceptions. Headline quality detection using established DistilBERT and bagging classifier models showed that quality misclassification was common, with detection accuracies only ranging from 35.2% to 63.5%. These findings suggest the following: tested SLMs generally are compliant in generating falsified headlines, although there are slight variations in ethical restraints, and the generated headlines did not closely resemble existing primarily human-written content on the web, given the low quality classification accuracy.