# Unequal Verdicts: Investigating Gender Bias in LLM-Based Fake News Detection

arXiv: https://arxiv.org/abs/2608.03627 · HTML: https://mlsanity.com/paper/2608.03627/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 46 references that are in the map.
- First posted on arXiv 4 Aug 2026.

## Builds on (most cited first, 21 of 21)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://mlsanity.com/paper/2209.07858/) (2022)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond](https://mlsanity.com/paper/2304.13712/) (2023)
- ["Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection](https://mlsanity.com/paper/1705.00648/) (2017)
- ["Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letters](https://mlsanity.com/paper/2310.09219/) (2023)
- [Gender bias and stereotypes in Large Language Models](https://mlsanity.com/paper/2308.14921/) (2023)
- [Combating Misinformation in the Age of LLMs: Opportunities and Challenges](https://mlsanity.com/paper/2311.05656/) (2023)
- [Fairlearn: Assessing and Improving Fairness of AI Systems](https://mlsanity.com/paper/2303.16626/) (2023)
- [Disclosure and Mitigation of Gender Bias in LLMs](https://mlsanity.com/paper/2402.11190/) (2024)
- [Contrastive Language-Vision AI Models Pretrained on Web-Scraped Multimodal Data Exhibit Sexual Objectification Bias](https://mlsanity.com/paper/2212.11261/) (2022)
- [Disinformation Detection: An Evolving Challenge in the Age of LLMs](https://mlsanity.com/paper/2309.15847/) (2023)
- [Bias of AI-Generated Content: An Examination of News Produced by Large Language Models](https://mlsanity.com/paper/2309.09825/) (2023)
- [Large Language Models are Biased Because They Are Large Language Models](https://mlsanity.com/paper/2406.13138/) (2024)
- [Fact-checking information from large language models can decrease headline discernment](https://mlsanity.com/paper/2308.10800/) (2023)
- [Unmasking Digital Falsehoods: A Comparative Analysis of LLM-Based Misinformation Detection Strategies](https://mlsanity.com/paper/2503.00724/) (2025)
- [Evaluating Gender Bias of LLMs in Making Morality Judgements](https://mlsanity.com/paper/2410.09992/) (2024)
- [GenderCARE: A Comprehensive Framework for Assessing and Reducing Gender Bias in Large Language Models](https://mlsanity.com/paper/2408.12494/) (2024)
- [Gender Bias in Machine Translation and The Era of Large Language Models](https://mlsanity.com/paper/2401.10016/) (2024)
- [AdvCat: Domain-Agnostic Robustness Assessment for Cybersecurity-Critical Applications with Categorical Inputs](https://mlsanity.com/paper/2212.13989/) (2022)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly used for automated fact-checking, yet their susceptibility to gender bias in this context remains underexplored. This study presents the first systematic investigation of gender bias in LLM-based fake news detection using real-world data. We augment the LIAR benchmark with three gender variants of speaker job titles (Neutral, Male, Female) for each statement to test whether veracity judgments vary solely based on gender presentation. Six state-of-the-art LLMs are evaluated across multiple bias and fairness metrics. All models exhibit gender sensitivity: 9.79%-35.13% of statements receive inconsistent labels across the three variants, with Male-Female comparisons showing 6.5%-23.6% flip rates. Two primary bias manifestations are identified: instability (inconsistent judgments) and directionality (systematic favoritism). Five models show statistically significant directional effects, with the strongest effects displaying male-skeptic patterns. These findings demonstrate that gender bias undermines both reliability and fairness in LLM-based fake news detection, highlighting the need for bias-aware evaluation and mitigation strategies. The augmented dataset is publicly released to support future research.