# Large Language Models are Perplexed by some Political Parties

arXiv: https://arxiv.org/abs/2606.05937 · HTML: https://mlsanity.com/paper/2606.05937/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 50 references that are in the map.
- First posted on arXiv 4 Jun 2026.

## Builds on (most cited first, 16 of 16)

- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models](https://mlsanity.com/paper/2305.08283/) (2023)
- [The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation](https://mlsanity.com/paper/2301.01768/) (2023)
- [Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics](https://mlsanity.com/paper/2106.14574/) (2021)
- [Hidden Persuaders: LLMs' Political Leaning and Their Influence on Voters](https://mlsanity.com/paper/2410.24190/) (2024)
- [Beyond prompt brittleness: Evaluating the reliability and consistency of political worldviews in LLMs](https://mlsanity.com/paper/2402.17649/) (2024)
- [Are All Languages Equally Hard to Language-Model?](https://mlsanity.com/paper/1806.03743/) (2018)
- [Opportunities and Risks of LLMs for Scalable Deliberation with Polis](https://mlsanity.com/paper/2306.11932/) (2023)
- [IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistance](https://mlsanity.com/paper/2502.08395/) (2025)
- [Do Large Language Models Have an English Accent? Evaluating and Improving the Naturalness of Multilingual LLMs](https://mlsanity.com/paper/2410.15956/) (2024)
- [What Is The Political Content in LLMs' Pre- and Post-Training Data?](https://mlsanity.com/paper/2509.22367/) (2025)
- [Do Political Opinions Transfer Between Western Languages? An Analysis of Unaligned and Aligned Multilingual LLMs](https://mlsanity.com/paper/2508.05553/) (2025)
- [Polyglots or Multitudes? Multilingual LLM Answers to Value-laden Multiple-Choice Questions](https://mlsanity.com/paper/2602.05932/) (2026)
- [Equalized Generative Treatment: Matching f-divergences for Fairness in Generative Models](https://mlsanity.com/paper/2602.08660/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly used, including in political applications, but their political fairness has been little studied. We assess it using perplexity, posing that a fair model should give equal probability to all political groups. However, we find, across ten LLMs and three datasets covering 37 languages, that LLMs are more perplexed by the texts of far right and nationalist parties than of social-democratic parties. We find this to be consistent with previous work on translation fairness, to the point that perplexity correlates with downstream translation metrics. Our method is applicable to both base LLMs as well as their instruction-tuned counterpart, and we find that both are highly correlated, suggesting that the political fairness of LLMs stems from their pretraining, and is hardly affected by instruction-tuning.