# A Detailed Factor Analysis for the Political Compass Test: Navigating Ideologies of Large Language Models

arXiv: https://arxiv.org/abs/2506.22493 · HTML: https://mlsanity.com/paper/2506.22493/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 32 references that are in the map.
- First posted on arXiv 24 Jun 2025.

## Builds on (most cited first, 17 of 17)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies](https://mlsanity.com/paper/1804.11283/) (2018)
- [From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models](https://mlsanity.com/paper/2305.08283/) (2023)
- [The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation](https://mlsanity.com/paper/2301.01768/) (2023)
- [Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models](https://mlsanity.com/paper/2402.16786/) (2024)
- [We Can Detect Your Bias: Predicting the Political Ideology of News Articles](https://mlsanity.com/paper/2010.05338/) (2020)
- [Measuring Political Bias in Large Language Models: What Is Said and How It Is Said](https://mlsanity.com/paper/2403.18932/) (2024)
- [ScisummNet: A Large Annotated Corpus and Content-Impact Models for Scientific Paper Summarization with Citation Networks](https://mlsanity.com/paper/1909.01716/) (2019)
- [Hidden Persuaders: LLMs' Political Leaning and Their Influence on Voters](https://mlsanity.com/paper/2410.24190/) (2024)
- [Large Language Model Psychometrics: A Systematic Review of Evaluation, Validation, and Enhancement](https://mlsanity.com/paper/2505.08245/) (2025)
- [How Susceptible are Large Language Models to Ideological Manipulation?](https://mlsanity.com/paper/2402.11725/) (2024)
- [Reading Between the Tweets: Deciphering Ideological Stances of Interconnected Mixed-Ideology Communities](https://mlsanity.com/paper/2402.01091/) (2024)
- [The Political Ideology of Large Language Models: Measurement, Inconsistency, and Persuasive Influence](https://mlsanity.com/paper/2505.04171/) (2025)

## Built on by (most cited first, 3 of 3)

- [The Shared Latent Structure of LLM Psychometric Responses](https://mlsanity.com/paper/2605.05080/) (2026)
- [LLM-Ideoplasticity: Measuring Ideological Plasticity in the Political Behavior of LLMs as a Context-Conditioned Distribution](https://mlsanity.com/paper/2606.28335/) (2026)
- [Navigating the digital spectrum: Assessing political bias, stability, and downstream fairness in Large Language Models](https://mlsanity.com/paper/2609.08637/) (2026)

## Abstract (arXiv)

The Political Compass Test (PCT) and similar surveys are commonly used to assess political bias in auto-regressive LLMs. Our rigorous statistical experiments show that while changes to standard generation parameters have minimal effect on PCT scores, prompt phrasing and fine-tuning individually and together can significantly influence results. Interestingly, fine-tuning on politically rich vs. neutral datasets does not lead to different shifts in scores. We also generalize these findings to a similar popular test called 8 Values. Humans do not change their responses to questions when prompted differently (``answer this question'' vs ``state your opinion''), or after exposure to politically neutral text, such as mathematical formulae. But the fact that the models do so raises concerns about the validity of these tests for measuring model bias, and paves the way for deeper exploration into how political and social views are encoded in LLMs.