# Assessing Political Bias in Large Language Models

arXiv: https://arxiv.org/abs/2405.13041 · HTML: https://mlsanity.com/paper/2405.13041/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 48 references that are in the map.
- First posted on arXiv 17 May 2024.

## Builds on (most cited first, 7 of 7)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [Persistent Anti-Muslim Bias in Large Language Models](https://mlsanity.com/paper/2101.05783/) (2021)
- [Performative Prediction](https://mlsanity.com/paper/2002.06673/) (2020)
- [Performative Power](https://mlsanity.com/paper/2203.17232/) (2022)

## Built on by (most cited first, 30 of 32)

- [PERSONA: A Reproducible Testbed for Pluralistic Alignment](https://mlsanity.com/paper/2407.17387/) (2024)
- [Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs](https://mlsanity.com/paper/2502.08640/) (2025)
- [IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistance](https://mlsanity.com/paper/2502.08395/) (2025)
- [When Your AI Agent Succumbs to Peer-Pressure: Studying Opinion-Change Dynamics of LLMs](https://mlsanity.com/paper/2510.19107/) (2025)
- [Beyond Partisan Leaning: A Comparative Analysis of Political Bias in Large Language Models](https://mlsanity.com/paper/2412.16746/) (2024)
- [A Simulation System Towards Solving Societal-Scale Manipulation](https://mlsanity.com/paper/2410.13915/) (2024)
- [Large Means Left: Political Bias in Large Language Models Increases with Their Number of Parameters](https://mlsanity.com/paper/2505.04393/) (2025)
- [What Large Language Models Do Not Talk About: An Empirical Study of Moderation and Censorship Practices](https://mlsanity.com/paper/2504.03803/) (2025)
- [In Agents We Trust, but Who Do Agents Trust? Latent Source Preferences Steer LLM Generations](https://mlsanity.com/paper/2602.15456/) (2026)
- [A Cautionary Tale About "Neutrally" Informative AI Tools Ahead of the 2025 Federal Elections in Germany](https://mlsanity.com/paper/2502.15568/) (2025)
- [SocialHarmBench: Revealing LLM Vulnerabilities to Socially Harmful Requests](https://mlsanity.com/paper/2510.04891/) (2025)
- [The Prompt War: How AI Decides on a Military Intervention](https://mlsanity.com/paper/2507.06277/) (2025)
- [On the Inevitability of Left-Leaning Political Bias in Aligned Language Models](https://mlsanity.com/paper/2507.15328/) (2025)
- [HULLMI: Human vs LLM identification with explainability](https://mlsanity.com/paper/2409.04808/) (2024)
- [Adaptive political surveys and GPT-4: Tackling the cold start problem with simulated user interactions](https://mlsanity.com/paper/2503.09311/) (2025)
- [Think Again! The Effect of Test-Time Compute on Preferences, Opinions, and Beliefs of Large Language Models](https://mlsanity.com/paper/2505.19621/) (2025)
- [How Similar Are Grokipedia and Wikipedia? A Multi-Dimensional Textual and Structural Comparison](https://mlsanity.com/paper/2510.26899/) (2025)
- [Don't Change My View: Ideological Bias Auditing in Large Language Models](https://mlsanity.com/paper/2509.12652/) (2025)
- [The Language You Ask In: Language-Conditioned Ideological Divergence in LLM Analysis of Contested Political Documents](https://mlsanity.com/paper/2601.12164/) (2026)
- [Black Box Deployed -- Functional Criteria for Artificial Moral Agents in the LLM Era](https://mlsanity.com/paper/2507.13175/) (2025)
- [Fair Representation in Parliamentary Summaries: Measuring and Mitigating Inclusion Bias](https://mlsanity.com/paper/2507.14221/) (2025)
- [Left Leaning Models: How AI Evaluates Economic Policy?](https://mlsanity.com/paper/2507.15771/) (2025)
- [LLM Analysis of 150+ years of German Parliamentary Debates on Migration Reveals Shift from Post-War Solidarity to Anti-Solidarity in the Last Decade](https://mlsanity.com/paper/2509.07274/) (2025)
- [Measuring Algorithmic Partisanship via Zero-Shot Classification and Its Implications on Political Discourse](https://mlsanity.com/paper/2510.01258/) (2025)
- [Measuring Political Stance and Consistency in Large Language Models](https://mlsanity.com/paper/2601.17016/) (2026)
- [Hidden Topics: Measuring Sensitive AI Beliefs with List Experiments](https://mlsanity.com/paper/2602.21939/) (2026)
- [Multi-Agent Strategic Games with LLMs](https://mlsanity.com/paper/2605.03604/) (2026)
- [Detecting undisclosed LLM-generated content in parliamentary texts](https://mlsanity.com/paper/2606.14209/) (2026)
- [!Imperio, smolVLA: The Implications of Data Poisoning on Open Source Robotics](https://mlsanity.com/paper/2607.04146/) (2026)
- [Directional Hallucinations: Ideological Drift in News-Grounded LLM Question Answering](https://mlsanity.com/paper/2607.20487/) (2026)

## Abstract (arXiv)

The assessment of bias within Large Language Models (LLMs) has emerged as a critical concern in the contemporary discourse surrounding Artificial Intelligence (AI) in the context of their potential impact on societal dynamics. Recognizing and considering political bias within LLM applications is especially important when closing in on the tipping point toward performative prediction. Then, being educated about potential effects and the societal behavior LLMs can drive at scale due to their interplay with human operators. In this way, the upcoming elections of the European Parliament will not remain unaffected by LLMs. We evaluate the political bias of the currently most popular open-source LLMs (instruct or assistant models) concerning political issues within the European Union (EU) from a German voter's perspective. To do so, we use the "Wahl-O-Mat," a voting advice application used in Germany. From the voting advice of the "Wahl-O-Mat" we quantize the degree of alignment of LLMs with German political parties. We show that larger models, such as Llama3-70B, tend to align more closely with left-leaning political parties, while smaller models often remain neutral, particularly when prompted in English. The central finding is that LLMs are similarly biased, with low variances in the alignment concerning a specific party. Our findings underline the importance of rigorously assessing and making bias transparent in LLMs to safeguard the integrity and trustworthiness of applications that employ the capabilities of performative prediction and the invisible hand of machine learning prediction and language generation.