# Only a Little to the Left: A Theory-grounded Measure of Political Bias in Large Language Models

arXiv: https://arxiv.org/abs/2503.16148 · HTML: https://mlsanity.com/paper/2503.16148/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 67 references that are in the map.
- First posted on arXiv 20 Mar 2025.

## Builds on (most cited first, 29 of 29)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models](https://mlsanity.com/paper/2305.08283/) (2023)
- [Large Language Models in Finance: A Survey](https://mlsanity.com/paper/2311.10723/) (2023)
- [The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation](https://mlsanity.com/paper/2301.01768/) (2023)
- [Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models](https://mlsanity.com/paper/2402.16786/) (2024)
- [Ask Me Anything: A simple strategy for prompting language models](https://mlsanity.com/paper/2210.02441/) (2022)
- [Probing Pre-Trained Language Models for Cross-Cultural Differences in Values](https://mlsanity.com/paper/2203.13722/) (2022)
- [Measuring Political Bias in Large Language Models: What Is Said and How It Is Said](https://mlsanity.com/paper/2403.18932/) (2024)
- [The Political Preferences of LLMs](https://mlsanity.com/paper/2402.01789/) (2024)
- [The Self-Perception and Political Biases of ChatGPT](https://mlsanity.com/paper/2304.07333/) (2023)
- [Beyond prompt brittleness: Evaluating the reliability and consistency of political worldviews in LLMs](https://mlsanity.com/paper/2402.17649/) (2024)
- [You don't need a personality test to know these models are unreliable: Assessing the Reliability of Large Language Models on Psychometric Instruments](https://mlsanity.com/paper/2311.09718/) (2023)
- [MEGAnno+: A Human-LLM Collaborative Annotation System](https://mlsanity.com/paper/2402.18050/) (2024)
- [Modeling Framing in Immigration Discourse on Social Media](https://mlsanity.com/paper/2104.06443/) (2021)
- [Fine-tuning and Utilization Methods of Domain-specific LLMs](https://mlsanity.com/paper/2401.02981/) (2024)
- [IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistance](https://mlsanity.com/paper/2502.08395/) (2025)
- [Exploring the Relationship between LLM Hallucinations and Prompt Linguistic Nuances: Readability, Formality, and Concreteness](https://mlsanity.com/paper/2309.11064/) (2023)
- [AI in the Gray: Exploring Moderation Policies in Dialogic Large Language Models vs. Human Answers in Controversial Topics](https://mlsanity.com/paper/2308.14608/) (2023)
- [Multilingual Coarse Political Stance Classification of Media. The Editorial Line of a ChatGPT and Bard Newspaper](https://mlsanity.com/paper/2310.16269/) (2023)

## Built on by (most cited first, 15 of 15)

- [Replicating Human Motivated Reasoning Studies with LLMs](https://mlsanity.com/paper/2601.16130/) (2026)
- [On the Inevitability of Left-Leaning Political Bias in Aligned Language Models](https://mlsanity.com/paper/2507.15328/) (2025)
- [Apparent Psychological Profiles of Large Language Models are Largely a Measurement Artifact](https://mlsanity.com/paper/2606.20205/) (2026)
- [Position: Universal Aesthetic Alignment Narrows Artistic Expression](https://mlsanity.com/paper/2512.11883/) (2025)
- [The Proxy Presumption: From Semantic Embeddings to Valid Social Measures](https://mlsanity.com/paper/2605.07409/) (2026)
- [Left Leaning Models: How AI Evaluates Economic Policy?](https://mlsanity.com/paper/2507.15771/) (2025)
- [Moral Lenses, Political Coordinates: Towards Ideological Positioning of Morally Conditioned LLMs](https://mlsanity.com/paper/2601.08634/) (2026)
- [Hidden Topics: Measuring Sensitive AI Beliefs with List Experiments](https://mlsanity.com/paper/2602.21939/) (2026)
- [Multi-Agent Strategic Games with LLMs](https://mlsanity.com/paper/2605.03604/) (2026)
- [Progressive in Principle, Centrist in Practice: LLM Political Bias Is Instrument-Dependent](https://mlsanity.com/paper/2606.00048/) (2026)
- [LLM-Ideoplasticity: Measuring Ideological Plasticity in the Political Behavior of LLMs as a Context-Conditioned Distribution](https://mlsanity.com/paper/2606.28335/) (2026)
- [VIBE: A VAD-Informed Benchmark for Entity-Centered Affective Profiling of Large Language Model Outputs](https://mlsanity.com/paper/2608.03810/) (2026)
- [Who Would You Vote For? Auditing Political Alignment in LLMs: An Italian Case-Study](https://mlsanity.com/paper/2608.11649/) (2026)
- [We're Cooked! - Probing LLM Political Alignment Via Conflict-Framed Recipe Translation](https://mlsanity.com/paper/2609.07568/) (2026)
- [Navigating the digital spectrum: Assessing political bias, stability, and downstream fairness in Large Language Models](https://mlsanity.com/paper/2609.08637/) (2026)

## Abstract (arXiv)

Prompt-based language models like GPT4 and LLaMa have been used for a wide variety of use cases such as simulating agents, searching for information, or for content analysis. For all of these applications and others, political biases in these models can affect their performance. Several researchers have attempted to study political bias in language models using evaluation suites based on surveys, such as the Political Compass Test (PCT), often finding a particular leaning favored by these models. However, there is some variation in the exact prompting techniques, leading to diverging findings, and most research relies on constrained-answer settings to extract model responses. Moreover, the Political Compass Test is not a scientifically valid survey instrument. In this work, we contribute a political bias measured informed by political science theory, building on survey design principles to test a wide variety of input prompts, while taking into account prompt sensitivity. We then prompt 11 different open and commercial models, differentiating between instruction-tuned and non-instruction-tuned models, and automatically classify their political stances from 88,110 responses. Leveraging this dataset, we compute political bias profiles across different prompt variations and find that while PCT exaggerates bias in certain models like GPT3.5, measures of political bias are often unstable, but generally more left-leaning for instruction-tuned models. Code and data are available on: https://github.com/MaFa211/theory_grounded_pol_bias