Are LLMs More Skeptical of Entertainment News?
Not on the map. Why?
Abstract (arXiv)
Large language models (LLMs) are increasingly used for automated news credibility assessment, yet it remains unclear whether they apply even-handed standards across journalistic genres. We examine whether zero-shot LLMs are more likely to misclassify legitimate entertainment news as fake than legitimate hard news, using a within-dataset design on GossipCop from FakeNewsNet. Across four frontier models, we find a clear but model-specific genre asymmetry: DeepSeek-V3.2 and GPT-5.2 show false-positive-rate gaps of 10.1 and 8.8 percentage points, respectively (both p < .001), whereas Claude Opus 4.6 and Gemini 3 Flash show no comparable difference. A style-swap experiment yields only limited and inconsistent changes, suggesting that the asymmetry is not reducible to stylistic register alone. Prompt-based mitigation is likewise possible but not generic: framing the model as an entertainment-news fact-checker reduces false positives for DeepSeek-V3.2 by about 50% without detectable recall loss, but offers little improvement for GPT-5.2. Exploratory qualitative coding further suggests two recurring error patterns in sampled false positives: treating private-life claims as inherently unverifiable and discounting entertainment journalism as an epistemically weaker genre. Taken together, these findings show that aggregate performance metrics can obscure structured false positives within legitimate journalism. We argue that LLM-based credibility assessment may not only evaluate truth claims but also differentially recognize the legitimacy of journalistic genres, and that evaluation should therefore include genre-stratified false-positive analysis alongside overall accuracy.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 25 of its 51 references that are in the map. First posted on arXiv 3 May 2026.
Builds on
25 of 51 references.
References · 25
- 2019A Survey on Bias and Fairness in Machine Learningcited by 1,917
- 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
- 2018FakeNewsNet: A Data Repository with News Content, Social Context and Spatialtemporal Information for Studying Fake News on Social Mediacited by 285
- 2024ProSA: Assessing and Understanding the Prompt Sensitivity of LLMscited by 149
- 2017Automatic Detection of Fake Newscited by 145
- 2023The science of fake newscited by 97
- 2017A Stylometric Inquiry into Hyperpartisan and Fake Newscited by 82
- 2023Bad Actor, Good Advisor: Exploring the Role of Large Language Models in Fake News Detectioncited by 72
- 2023Fake News in Sheep's Clothing: Robust Fake News Detection Against LLM-Empowered Style Attackscited by 70
- 2019The Limitations of Stylometry for Detecting Machine-Generated Fake Newscited by 63
- 2017This Just In: Fake News Packs a Lot in Title, Uses Simpler, Repetitive Content in Text Body, More Similar to Satire than Real Newscited by 59
- 2023Towards Reliable Misinformation Mitigation: Generalization, Uncertainty, and GPT-4cited by 45
13 more
- 2025The simulation of judgment in LLMscited by 22
- 2021A Review of Text Style Transfer using Deep Learningcited by 18
- 2023Fake News Detectors are Biased against Texts Generated by Large Language Modelscited by 18
- 2023Weakly Supervised Veracity Classification with LLM-Predicted Credibility Signalscited by 14
- 2023Adapting Fake News Detection to the Era of Large Language Modelscited by 14
- 2023Explore the Potential of LLMs in Misinformation Detection: An Empirical Studycited by 12
- 2024Comparing GPT-4 and Open-Source Language Models in Misinformation Mitigationcited by 9
- 2024Can Large Language Models Detect Misinformation in Scientific News Reporting?cited by 9
- 2025Truth over Tricks: Measuring and Mitigating Shortcut Learning in Misinformation Detectioncited by 8
- 2026Robust Fake News Detection using Large Language Models under Adversarial Sentiment Attackscited by 8
- 2023SoK: Machine Learning for Misinformation Detectioncited by 4
- 2024SC2: Towards Enhancing Content Preservation and Style Consistency in Long Text Style Transfercited by 4
- 2024Variation between Credible and Non-Credible News Across Topicscited by 1
Built on by
Not cited yet.