# Analyzing Political Bias in LLMs via Target-Oriented Sentiment Classification

arXiv: https://arxiv.org/abs/2505.19776 · HTML: https://mlsanity.com/paper/2505.19776/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 48 references that are in the map.
- First posted on arXiv 26 May 2025.

## Builds on (most cited first, 23 of 23)

- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [Sentiment Analysis in the Era of Large Language Models: A Reality Check](https://mlsanity.com/paper/2305.15005/) (2023)
- [Evaluating the Moral Beliefs Encoded in LLMs](https://mlsanity.com/paper/2307.14324/) (2023)
- [Societal Biases in Language Generation: Progress and Challenges](https://mlsanity.com/paper/2105.04054/) (2021)
- [The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation](https://mlsanity.com/paper/2301.01768/) (2023)
- [Reducing Sentiment Bias in Language Models via Counterfactual Evaluation](https://mlsanity.com/paper/1911.03064/) (2019)
- [Investigating Cultural Alignment of Large Language Models](https://mlsanity.com/paper/2402.13231/) (2024)
- [Large Language Models Meet NLP: A Survey](https://mlsanity.com/paper/2405.12819/) (2024)
- [SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoning](https://mlsanity.com/paper/2309.04766/) (2023)
- [Measuring Political Bias in Large Language Models: What Is Said and How It Is Said](https://mlsanity.com/paper/2403.18932/) (2024)
- [Moral Foundations of Large Language Models](https://mlsanity.com/paper/2310.15337/) (2023)
- [A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias](https://mlsanity.com/paper/2404.00929/) (2024)
- [Fine-Tuned 'Small' LLMs (Still) Significantly Outperform Zero-Shot Generative AI Models in Text Classification](https://mlsanity.com/paper/2406.08660/) (2024)
- [Large Language Models Reflect the Ideology of their Creators](https://mlsanity.com/paper/2410.18417/) (2024)
- [Detecting Hate Speech with GPT-3](https://mlsanity.com/paper/2103.12407/) (2021)
- [Ethical Reasoning and Moral Value Alignment of LLMs Depend on the Language we Prompt them in](https://mlsanity.com/paper/2404.18460/) (2024)
- [The Shifted and The Overlooked: A Task-oriented Investigation of User-GPT Interactions](https://mlsanity.com/paper/2310.12418/) (2023)
- [The Unequal Opportunities of Large Language Models: Revealing Demographic Bias through Job Recommendations](https://mlsanity.com/paper/2308.02053/) (2023)
- [Speaking Multiple Languages Affects the Moral Bias of Language Models](https://mlsanity.com/paper/2211.07733/) (2022)
- [Combining Objective and Subjective Perspectives for Political News Understanding](https://mlsanity.com/paper/2408.11174/) (2024)

## Built on by (most cited first, 7 of 7)

- [Read Between the Lines: A Benchmark for Uncovering Political Bias in Bangla News Articles](https://mlsanity.com/paper/2510.03898/) (2025)
- [A Scalable Entity-Based Framework for Auditing Bias in Large Language Models](https://mlsanity.com/paper/2601.12374/) (2026)
- [Left, Right, or Center? Evaluating LLM Framing in News Classification and Generation](https://mlsanity.com/paper/2601.05835/) (2026)
- [Measuring Algorithmic Partisanship via Zero-Shot Classification and Its Implications on Political Discourse](https://mlsanity.com/paper/2510.01258/) (2025)
- [A Detailed Study on LLM Biases Concerning Corporate Social Responsibility and Green Supply Chains](https://mlsanity.com/paper/2511.01840/) (2025)
- [From Entity Mentions to Tone: An LLM-Based Pipeline for Media Bias Analysis](https://mlsanity.com/paper/2608.17454/) (2026)
- [Navigating the digital spectrum: Assessing political bias, stability, and downstream fairness in Large Language Models](https://mlsanity.com/paper/2609.08637/) (2026)

## Abstract (arXiv)

Political biases encoded by LLMs might have detrimental effects on downstream applications. Existing bias analysis methods rely on small-size intermediate tasks (questionnaire answering or political content generation) and rely on the LLMs themselves for analysis, thus propagating bias. We propose a new approach leveraging the observation that LLM sentiment predictions vary with the target entity in the same sentence. We define an entropy-based inconsistency metric to encode this prediction variability. We insert 1319 demographically and politically diverse politician names in 450 political sentences and predict target-oriented sentiment using seven models in six widely spoken languages. We observe inconsistencies in all tested combinations and aggregate them in a statistically robust analysis at different granularity levels. We observe positive and negative bias toward left and far-right politicians and positive correlations between politicians with similar alignment. Bias intensity is higher for Western languages than for others. Larger models exhibit stronger and more consistent biases and reduce discrepancies between similar languages. We partially mitigate LLM unreliability in target-oriented sentiment classification (TSC) by replacing politician names with fictional but plausible counterparts.