mlsanityData to 7 Oct 2026

Beyond prompt brittleness: Evaluating the reliability and consistency of political worldviews in LLMs

arXiv 2402.17649 ↗First posted 27 Feb 2024cs.CL cs.CYcited by 55

Not on the map. Why?

Abstract (arXiv)

Due to the widespread use of large language models (LLMs), we need to understand whether they embed a specific "worldview" and what these views reflect. Recent studies report that, prompted with political questionnaires, LLMs show left-liberal leanings (Feng et al., 2023; Motoki et al., 2024). However, it is as yet unclear whether these leanings are reliable (robust to prompt variations) and whether the leaning is consistent across policies and political leaning. We propose a series of tests which assess the reliability and consistency of LLMs' stances on political statements based on a dataset of voting-advice questionnaires collected from seven EU countries and annotated for policy issues. We study LLMs ranging in size from 7B to 70B parameters and find that their reliability increases with parameter count. Larger models show overall stronger alignment with left-leaning parties but differ among policy programs: They show a (left-wing) positive stance towards environment protection, social welfare state and liberal society but also (right-wing) law and order, with no consistent preferences in the areas of foreign policy and migration.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Builds on 31 of its 73 references that are in the map. First posted on arXiv 27 Feb 2024.

Builds on

31 of 73 references.

References · 31

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Mistral 7Bcited by 3,065
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  6. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  8. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  9. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  10. 2023Whose Opinions Do Language Models Reflect?cited by 759
  11. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  12. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
19 more
  1. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  2. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 299
  3. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 257
  4. 2019A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cyclecited by 192
  5. 2023Gender bias and stereotypes in Large Language Modelscited by 185
  6. 2023The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientationcited by 183
  7. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 176
  8. 2022Probing Pre-Trained Language Models for Cross-Cultural Differences in Valuescited by 147
  9. 2023Questioning the Survey Responses of Large Language Modelscited by 70
  10. 2023The Self-Perception and Political Biases of ChatGPTcited by 66
  11. 2021Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Promptscited by 64
  12. 2023ROBBIE: Robust Bias Evaluation of Large Generative Language Modelscited by 53
  13. 2023You don't need a personality test to know these models are unreliable: Assessing the Reliability of Large Language Models on Psychometric Instrumentscited by 48
  14. 2023Not All Countries Celebrate Thanksgiving: On the Cultural Dominance in Large Language Modelscited by 42
  15. 2023Evaluating Open-QA Evaluationcited by 35
  16. 2022Spurious Correlations in Reference-Free Evaluation of Text Generationcited by 26
  17. 2023''Fifty Shades of Bias'': Normative Ratings of Gender Bias in GPT Generated English Textcited by 15
  18. 2023Simple Linguistic Inferences of Large Language Models (LLMs): Blind Spots and Blindscited by 7
  19. 2023Additive manifesto decomposition: A policy domain aware method for understanding party positioningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 19; 2026: 25.

55 papers.

Sort:

By area:

  1. 2024Measuring Political Bias in Large Language Models: What Is Said and How It Is Saidcited by 89
  2. 2024Benchmarking Distributional Alignment of Large Language Modelscited by 58
  3. 2024Take Caution in Using LLMs as Human Surrogates: Scylla Ex Machinacited by 29
  4. 2024The Potential and Challenges of Evaluating Attitudes, Opinions, and Values in Large Language Modelscited by 19
  5. 2024Real or Robotic? Assessing Whether LLMs Accurately Simulate Qualities of Human Responses in Dialoguecited by 15
  6. 2025Only a Little to the Left: A Theory-grounded Measure of Political Bias in Large Language Modelscited by 15
  7. 2024Which Demographics do LLMs Default to During Annotation?cited by 10
  8. 2025LLM Knowledge is Brittle: Truthfulness Representations Rely on Superficial Resemblancecited by 9
  9. 2026One Persona, Many Cues, Different Results: How Sociodemographic Cues Impact LLM Personalizationcited by 9
  10. 2025What Is The Political Content in LLMs' Pre- and Post-Training Data?cited by 7
  11. 2025Do Political Opinions Transfer Between Western Languages? An Analysis of Unaligned and Aligned Multilingual LLMscited by 6
  12. 2024Revealing Fine-Grained Values and Opinions in Large Language Modelscited by 5
38 more
  1. 2025Framing Political Bias in Multilingual LLMs Across Pakistani Languagescited by 5
  2. 2025Multilingual Political Views of Large Language Models: Identification and Steeringcited by 5
  3. 2025Political Ideology Shifts in Large Language Modelscited by 5
  4. 2026Political Alignment in Large Language Models: A Multidimensional Audit of Psychometric Identity and Behavioral Biascited by 5
  5. 2025Generalizability of Media Frames: Corpus creation and analysis across countriescited by 4
  6. 2025Psychometric Personality Shaping Modulates Capabilities and Safety in Language Modelscited by 4
  7. 2026Beyond Marginal Distributions: A Framework to Evaluate the Representativeness of Demographic-Aligned LLMscited by 4
  8. 2025On the Inevitability of Left-Leaning Political Bias in Aligned Language Modelscited by 3
  9. 2026AI-exhibited Personality Traits Can Shape Human Self-concept through Conversationscited by 3
  10. 2026On the Credibility of Evaluating LLMs using Survey Questionscited by 3
  11. 2026Political Bias Audits of LLMs Capture Sycophancy to the Inferred Auditorcited by 3
  12. 2024Inertia in Moral and Value Judgments of Large Language Modelscited by 2
  13. 2025Leveraging In-Context Learning for Political Bias Testing of LLMscited by 2
  14. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  15. 2024A University Framework for the Responsible use of Generative AI in Researchcited by 1
  16. 2024Superhuman performance in urology board questions by an explainable large language model enabled for context integration of the European Association of Urology guidelines: the UroBot studycited by 1
  17. 2025PReSS: An Automated Black-Box Framework for Evaluating Political Stance Stability in LLMscited by 1
  18. 2025Don't Change My View: Ideological Bias Auditing in Large Language Modelscited by 1
  19. 2026Polyglots or Multitudes? Multilingual LLM Answers to Value-laden Multiple-Choice Questionscited by 1
  20. 2026Structuring the Space of Perspectivescited by 1
  21. 2026Effects of Answer Format Variation on Gender Bias in Large Language Modelscited by 1
  22. 2024A Few Hypocrites: Few-Shot Learning and Subtype Definitions for Detecting Hypocrisy Accusations in Online Climate Change Debatescited by 0
  23. 2025Splits! Flexible Sociocultural Linguistic Investigation at Scalecited by 0
  24. 2025Analyzing Cognitive Differences Among Large Language Models through the Lens of Social Worldviewcited by 0
  25. 2025Are Humans as Brittle as Large Language Models?cited by 0
  26. 2025Assessing the Political Fairness of Multilingual LLMs: A Case Study based on a 21-way Multiparallel EuroParl Datasetcited by 0
  27. 2025Who Sees the Risk? Stakeholder Conflicts and Explanatory Policies in LLM-based Risk Assessmentcited by 0
  28. 2025Knowing Your Uncertainty -- On the application of LLM in social sciencescited by 0
  29. 2026Moral Lenses, Political Coordinates: Towards Ideological Positioning of Morally Conditioned LLMscited by 0
  30. 2026Robust Bias Evaluation with FilBBQ: A Filipino Bias Benchmark for Question-Answering Language Modelscited by 0
  31. 2026Autoscoring Anticlimax: A Meta-analytic Understanding of AI's Short-answer Shortcomings and Wording Weaknessescited by 0
  32. 2026Progressive in Principle, Centrist in Practice: LLM Political Bias Is Instrument-Dependentcited by 0
  33. 2026Large Language Models are Perplexed by some Political Partiescited by 0
  34. 2026AuAu: A Benchmark for Auditing Authoritarian Alignment in Large Language Modelscited by 0
  35. 2026LLM-Ideoplasticity: Measuring Ideological Plasticity in the Political Behavior of LLMs as a Context-Conditioned Distributioncited by 0
  36. 2026Rethinking Indic AI from a Lens of Cultural Heritage Preservationcited by 0
  37. 2026Are LLMs ready for HardChoices?cited by 0
  38. 2026Understanding the Impact of Linguistic Realization Choices on LLM Stance with Causal Tracingcited by 0

50 most cited of 55 All 55 (JSON)