mlsanityData to 14 Oct 2026

A Detailed Factor Analysis for the Political Compass Test: Navigating Ideologies of Large Language Models

arXiv 2506.22493 ↗First posted 24 Jun 2025cs.CY cs.CL cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

The Political Compass Test (PCT) and similar surveys are commonly used to assess political bias in auto-regressive LLMs. Our rigorous statistical experiments show that while changes to standard generation parameters have minimal effect on PCT scores, prompt phrasing and fine-tuning individually and together can significantly influence results. Interestingly, fine-tuning on politically rich vs. neutral datasets does not lead to different shifts in scores. We also generalize these findings to a similar popular test called 8 Values. Humans do not change their responses to questions when prompted differently (``answer this question'' vs ``state your opinion''), or after exposure to politically neutral text, such as mathematical formulae. But the fact that the models do so raises concerns about the validity of these tests for measuring model bias, and paves the way for deeper exploration into how political and social views are encoded in LLMs.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 17 of its 32 references that are in the map. First posted on arXiv 24 Jun 2025.

Builds on

17 of 32 references.

References · 17

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  4. 2023Whose Opinions Do Language Models Reflect?cited by 773
  5. 2023The Falcon Series of Open Language Modelscited by 535
  6. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 294
  7. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 260
  8. 2023The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientationcited by 189
  9. 2024Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Modelscited by 162
  10. 2020We Can Detect Your Bias: Predicting the Political Ideology of News Articlescited by 92
  11. 2024Measuring Political Bias in Large Language Models: What Is Said and How It Is Saidcited by 89
  12. 2019ScisummNet: A Large Annotated Corpus and Content-Impact Models for Scientific Paper Summarization with Citation Networkscited by 73
5 more
  1. 2024Hidden Persuaders: LLMs' Political Leaning and Their Influence on Voterscited by 71
  2. 2025Large Language Model Psychometrics: A Systematic Review of Evaluation, Validation, and Enhancementcited by 45
  3. 2024How Susceptible are Large Language Models to Ideological Manipulation?cited by 26
  4. 2024Reading Between the Tweets: Deciphering Ideological Stances of Interconnected Mixed-Ideology Communitiescited by 7
  5. 2025The Political Ideology of Large Language Models: Measurement, Inconsistency, and Persuasive Influencecited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026The Shared Latent Structure of LLM Psychometric Responsescited by 0
  2. 2026LLM-Ideoplasticity: Measuring Ideological Plasticity in the Political Behavior of LLMs as a Context-Conditioned Distributioncited by 0
  3. 2026Navigating the digital spectrum: Assessing political bias, stability, and downstream fairness in Large Language Modelscited by 0