# Quantifying Generative Media Bias with a Corpus of Real-world and Generated News Articles

arXiv: https://arxiv.org/abs/2406.10773 · HTML: https://mlsanity.com/paper/2406.10773/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 60 references that are in the map.
- First posted on arXiv 16 Jun 2024.

## Builds on (most cited first, 28 of 28)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- [Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies](https://mlsanity.com/paper/1804.11283/) (2018)
- [From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models](https://mlsanity.com/paper/2305.08283/) (2023)
- [We Can Detect Your Bias: Predicting the Political Ideology of News Articles](https://mlsanity.com/paper/2010.05338/) (2020)
- [On Decoding Strategies for Neural Text Generators](https://mlsanity.com/paper/2203.15721/) (2022)
- [A Closer Look at the Limitations of Instruction Tuning](https://mlsanity.com/paper/2402.05119/) (2024)
- [Neural Media Bias Detection Using Distant Supervision With BABE -- Bias Annotations By Experts](https://mlsanity.com/paper/2209.14557/) (2022)
- [Investigating Bias in LLM-Based Bias Detection: Disparities between LLMs and Human Perception](https://mlsanity.com/paper/2403.14896/) (2024)
- [POLITICS: Pretraining with Same-story Article Comparison for Ideology Prediction and Stance Detection](https://mlsanity.com/paper/2205.00619/) (2022)
- [Analyzing Political Bias and Unfairness in News Articles at Different Levels of Granularity](https://mlsanity.com/paper/2010.10652/) (2020)
- [Navigating the Complexity of Generative AI Adoption in Software Engineering](https://mlsanity.com/paper/2307.06081/) (2023)
- [Multi-view Models for Political Ideology Detection of News Articles](https://mlsanity.com/paper/1809.03485/) (2018)
- [KCD: Knowledge Walks and Textual Cues Enhanced Political Perspective Detection in News Media](https://mlsanity.com/paper/2204.04046/) (2022)
- [Friend or Foe? Exploring the Implications of Large Language Models on the Science System](https://mlsanity.com/paper/2306.09928/) (2023)
- [IndiVec: An Exploration of Leveraging Large Language Models for Media Bias Detection with Fine-Grained Bias Indicators](https://mlsanity.com/paper/2402.00345/) (2024)
- [KGAP: Knowledge Graph Augmented Political Perspective Detection in News Media](https://mlsanity.com/paper/2108.03861/) (2021)
- [Envisioning the Applications and Implications of Generative AI for News Media](https://mlsanity.com/paper/2402.18835/) (2024)
- [Multilingual Coarse Political Stance Classification of Media. The Editorial Line of a ChatGPT and Bard Newspaper](https://mlsanity.com/paper/2310.16269/) (2023)
- [T3L: Translate-and-Test Transfer Learning for Cross-Lingual Text Classification](https://mlsanity.com/paper/2306.04996/) (2023)
- [Evaluating Large Language Model Creativity from a Literary Perspective](https://mlsanity.com/paper/2312.03746/) (2023)

## Built on by (most cited first, 9 of 9)

- [IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistance](https://mlsanity.com/paper/2502.08395/) (2025)
- [LlamaLens: Specialized Multilingual LLM for Analyzing News and Social Media Content](https://mlsanity.com/paper/2410.15308/) (2024)
- [Mitigating Bias in RAG: Controlling the Embedder](https://mlsanity.com/paper/2502.17390/) (2025)
- [Multilingual Political Views of Large Language Models: Identification and Steering](https://mlsanity.com/paper/2507.22623/) (2025)
- [Neutralizing the Narrative: AI-Powered Debiasing of Online News Articles](https://mlsanity.com/paper/2504.03520/) (2025)
- [MGM: Global Understanding of Audience Overlap Graphs for Predicting the Factuality and the Bias of News Media](https://mlsanity.com/paper/2412.10467/) (2024)
- [Left, Right, or Center? Evaluating LLM Framing in News Classification and Generation](https://mlsanity.com/paper/2601.05835/) (2026)
- [Tracing Partisan Bias to Its Emotional Fingerprints: A Computational Approach to Mitigation](https://mlsanity.com/paper/2501.01284/) (2025)
- [Through the LLM Looking Glass: A Socratic Probing of Donkeys, Elephants, and Markets](https://mlsanity.com/paper/2503.16674/) (2025)

## Abstract (arXiv)

Large language models (LLMs) are increasingly being utilised across a range of tasks and domains, with a burgeoning interest in their application within the field of journalism. This trend raises concerns due to our limited understanding of LLM behaviour in this domain, especially with respect to political bias. Existing studies predominantly focus on LLMs undertaking political questionnaires, which offers only limited insights into their biases and operational nuances. To address this gap, our study establishes a new curated dataset that contains 2,100 human-written articles and utilises their descriptions to generate 56,700 synthetic articles using nine LLMs. This enables us to analyse shifts in properties between human-authored and machine-generated articles, with this study focusing on political bias, detecting it using both supervised models and LLMs. Our findings reveal significant disparities between base and instruction-tuned LLMs, with instruction-tuned models exhibiting consistent political bias. Furthermore, we are able to study how LLMs behave as classifiers, observing their display of political bias even in this role. Overall, for the first time within the journalistic domain, this study outlines a framework and provides a structured dataset for quantifiable experiments, serving as a foundation for further research into LLM political bias and its implications.