# When Neutral Summaries are not that Neutral: Quantifying Political Neutrality in LLM-Generated News Summaries

arXiv: https://arxiv.org/abs/2410.09978 · HTML: https://mlsanity.com/paper/2410.09978/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 59 references that are in the map.
- First posted on arXiv 13 Oct 2024.

## Builds on (most cited first, 20 of 20)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://mlsanity.com/paper/2102.01454/) (2021)
- [A Neural Attention Model for Abstractive Sentence Summarization](https://mlsanity.com/paper/1509.00685/) (2015)
- [From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models](https://mlsanity.com/paper/2305.08283/) (2023)
- [Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?](https://mlsanity.com/paper/2211.13972/) (2022)
- [Mitigating Political Bias in Language Models Through Reinforced Calibration](https://mlsanity.com/paper/2104.14795/) (2021)
- [Analyzing Polarization in Social Media: Method and Application to Tweets on 21 Mass Shootings](https://mlsanity.com/paper/1904.01596/) (2019)
- [POLITICS: Pretraining with Same-story Article Comparison for Ideology Prediction and Stance Detection](https://mlsanity.com/paper/2205.00619/) (2022)
- [A Framework for the Computational Linguistic Analysis of Dehumanization](https://mlsanity.com/paper/2003.03014/) (2020)
- [We Don't Speak the Same Language: Interpreting Polarization through Machine Translation](https://mlsanity.com/paper/2010.02339/) (2020)
- [KCD: Knowledge Walks and Textual Cues Enhanced Political Perspective Detection in News Media](https://mlsanity.com/paper/2204.04046/) (2022)
- [Vicarious Offense and Noise Audit of Offensive Speech Classifiers: Unifying Human and Machine Disagreement on What is Offensive](https://mlsanity.com/paper/2301.12534/) (2023)
- [Monoculture in Matching Markets](https://mlsanity.com/paper/2312.09841/) (2023)
- [The Silent Curriculum: How Does LLM Monoculture Shape Educational Content and Its Accessibility?](https://mlsanity.com/paper/2407.10371/) (2024)

## Built on by (most cited first, 7 of 7)

- [Can AI Truly Represent Your Voice in Deliberations? A Comprehensive Study of Large-Scale Opinion Aggregation with LLMs](https://mlsanity.com/paper/2510.05154/) (2025)
- [When Bigger Isn't Better: A Comprehensive Fairness Evaluation of Political Bias in Multi-News Summarisation](https://mlsanity.com/paper/2604.21309/) (2026)
- [Input Order Shapes LLM Semantic Alignment in Multi-Document Summarization](https://mlsanity.com/paper/2512.02665/) (2025)
- [DSIPA: Detecting LLM-Generated Texts via Sentiment-Invariant Patterns Divergence Analysis](https://mlsanity.com/paper/2604.26328/) (2026)
- [Benchmarking Gender and Political Bias in Large Language Models](https://mlsanity.com/paper/2509.06164/) (2025)
- [The Hidden Bias: A Study on Explicit and Implicit Political Stereotypes in Large Language Models](https://mlsanity.com/paper/2510.08236/) (2025)
- [Framing the Narrative: Ideological Mimicry in Large Language Models](https://mlsanity.com/paper/2609.38256/) (2026)

## Abstract (arXiv)

In an era where societal narratives are increasingly shaped by algorithmic curation, investigating the political neutrality of LLMs is an important research question. This study presents a fresh perspective on quantifying the political neutrality of LLMs through the lens of abstractive text summarization of polarizing news articles. We consider five pressing issues in current US politics: abortion, gun control/rights, healthcare, immigration, and LGBTQ+ rights. Via a substantial corpus of 20,344 news articles, our study reveals a consistent trend towards pro-Democratic biases in several well-known LLMs, with gun control and healthcare exhibiting the most pronounced biases (max polarization differences of -9.49% and -6.14%, respectively). Further analysis uncovers a strong convergence in the vocabulary of the LLM outputs for these divisive topics (55% overlap for Democrat-leaning representations, 52% for Republican). Being months away from a US election of consequence, we consider our findings important.