mlsanityData to 14 Oct 2026

Are Large Language Models Consistent over Value-laden Questions?

arXiv 2407.02996 ↗First posted 3 Jul 2024cs.CL cs.AIcited by 58

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) appear to bias their survey answers toward certain values. Nonetheless, some argue that LLMs are too inconsistent to simulate particular values. Are they? To answer, we first define value consistency as the similarity of answers across (1) paraphrases of one question, (2) related questions under one topic, (3) multiple-choice and open-ended use-cases of one question, and (4) multilingual translations of a question to English, Chinese, German, and Japanese. We apply these measures to small and large, open LLMs including llama-3, as well as gpt-4o, using 8,000 questions spanning more than 300 topics. Unlike prior work, we find that models are relatively consistent across paraphrases, use-cases, translations, and within a topic. Still, some inconsistencies remain. Models are more consistent on uncontroversial topics (e.g., in the U.S., "Thanksgiving") than on controversial ones ("euthanasia"). Base models are both more consistent compared to fine-tuned models and are uniform in their consistency across topics, while fine-tuned models are more inconsistent about some topics ("euthanasia") than others ("women's rights") like our human subjects (n=165).

Key facts. Cited by 58 papers in the map (data through 14 Oct 2026). Builds on 19 of its 33 references that are in the map. First posted on arXiv 3 Jul 2024.

Builds on

19 of 33 references.

References · 19

  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  4. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 437
  5. 2023Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Studycited by 206
  6. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
  7. 2023Understanding Social Reasoning in Language Models with Language Modelscited by 172
  8. 2024Towards Measuring and Modeling "Culture" in LLMs: A Surveycited by 167
  9. 2022Probing Pre-Trained Language Models for Cross-Cultural Differences in Valuescited by 149
  10. 2022Language Models as Agent Modelscited by 132
  11. 2023When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Taskscited by 90
  12. 2021Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefscited by 79
7 more
  1. 2024Auxiliary task demands mask the capabilities of smaller language modelscited by 54
  2. 2024Bias-Augmented Consistency Training Reduces Biased Reasoning in Chain-of-Thoughtcited by 39
  3. 2023Assessing LLMs for Moral Value Pluralismcited by 34
  4. 2023Challenging the Validity of Personality Tests for Large Language Modelscited by 25
  5. 2023What does ChatGPT return about human values? Exploring value bias in ChatGPT using a descriptive value theorycited by 19
  6. 2022Despite "super-human" performance, current LLMs are unsuited for decisions about ethics and safetycited by 13
  7. 2024The Echoes of Multilinguality: Tracing Cultural Value Shifts during LM Fine-tuningcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 30; 2026: 24.

58 papers.

Sort:

By area:

  1. 2025Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providerscited by 69
  2. 2025Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIscited by 67
  3. 2024Large Language Models Reflect the Ideology of their Creatorscited by 57
  4. 2025Values in the Wild: Discovering and Analyzing Values in Real-World Language Model Interactionscited by 52
  5. 2025The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMscited by 36
  6. 2026This human study did not involve human subjects: Validating LLM simulations as behavioral evidencecited by 29
  7. 2025IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistancecited by 26
  8. 2025Will AI Tell Lies to Save Sick Children? Litmus-Testing AI Values Prioritization with AIRiskDilemmascited by 18
  9. 2025What and Whose Knowledge? Measuring Epistemic Diversity in Large Language Modelscited by 14
  10. 2024Measuring Free-Form Decision-Making Inconsistency of Language Models in Military Crisis Simulationscited by 11
  11. 2025Prompt Stability Matters: Evaluating and Optimizing Auto-Generated Prompt in General-Purpose Systemscited by 10
  12. 2025Value Drifts: Tracing Value Alignment During LLM Post-Trainingcited by 10
38 more
  1. 2025Improving Language Model Personas via Rationalization with Psychological Scaffoldscited by 9
  2. 2025Large Language Models Often Say One Thing and Do Anothercited by 8
  3. 2025Revisiting LLM Value Probing Strategies: Are They Robust and Expressive?cited by 8
  4. 2025CoBRA: Programming Cognitive Bias in Social Agents Using Classic Social Science Experimentscited by 8
  5. 2025The Pluralistic Moral Gap: Understanding Judgment and Value Differences between Humans and Large Language Modelscited by 7
  6. 2026Knowing But Not Doing: Convergent Morality and Divergent Action in LLMscited by 7
  7. 2025Large Language Models for Interpretable Mental Health Diagnosiscited by 6
  8. 2025Evaluate Bias without Manual Test Sets: A Concept Representation Perspective for LLMscited by 6
  9. 2025Understanding and Mitigating Political Stance Cross-topic Generalization in Large Language Modelscited by 6
  10. 2025Systematic Bias in Large Language Models: Discrepant Response Patterns in Binary vs. Continuous Judgment Taskscited by 5
  11. 2025Evaluating LLM Adaptation to Sociodemographic Factors: User Profile vs. Dialogue Historycited by 5
  12. 2024SocialGaze: Improving the Integration of Human Social Norms in Large Language Modelscited by 4
  13. 2025Generative Psycho-Lexical Approach for Constructing Value Systems in Large Language Modelscited by 4
  14. 2025RankAlign: A Ranking View of the Generator-Validator Gap in Large Language Modelscited by 4
  15. 2025Isolating Culture Neurons in Multilingual Large Language Modelscited by 4
  16. 2026Human Values Matter: Investigating How Misalignment Shapes Collective Behaviors in LLM Agent Communitiescited by 4
  17. 2025Investigating Political and Demographic Associations in Large Language Models Through Moral Foundations Theorycited by 3
  18. 2026Reward Models Inherit Value Biases from Pretrainingcited by 3
  19. 2026On the Credibility of Evaluating LLMs using Survey Questionscited by 3
  20. 2026Are Language Models Sensitive to Morally Irrelevant Distractors?cited by 3
  21. 2025Who Gets the Kidney? Human-AI Alignment, Indecision, and Moral Valuescited by 2
  22. 2025Do Language Models Think Consistently? A Study of Value Preferences Across Varying Response Lengthscited by 2
  23. 2025Do Large Language Models Know Folktales? A Case Study of Yokai in Japanese Folktalescited by 2
  24. 2026Improving Cross-Format Robustness in Language Models with Multi-Format Trainingcited by 2
  25. 2026Incoherent Values? Probing LLM Preferences Through Parametric Variationcited by 2
  26. 2024Quantifying Risk Propensities of Large Language Models: Ethical Focus and Bias Detection through Role-Playcited by 1
  27. 2025MillStone: How Open-Minded Are LLMs?cited by 1
  28. 2025Implicit Values Embedded in How Humans and LLMs Complete Subjective Everyday Taskscited by 1
  29. 2025Reward Model Perspectives: Whose Opinions Do Reward Models Reward?cited by 1
  30. 2026How AI Systems Think About Education: Analyzing Latent Preference Patterns in Large Language Modelscited by 1
  31. 2026Do LLMs have core beliefs?cited by 1
  32. 2026People Are Not Just Their Countries. Disentangling Social Determinants of LLM Value Alignment Across Europecited by 1
  33. 2025VAL-Bench: Belief Consistency as a measure for Value Alignment in Language Modelscited by 0
  34. 2026Learning to Diagnose and Correct Moral Errors: Beyond Shallow Heuristics in Moral Alignmentcited by 0
  35. 2026Event-Centric Human Value Understanding in News-Domain Texts: An Actor-Conditioned, Multi-Granularity Benchmarkcited by 0
  36. 2026Teaching Values to Machines: Simulating Human-Like Behavior in LLMscited by 0
  37. 2026A Model of Multi-turn Human Persuadability Using Probabilistic Belief Tracingcited by 0
  38. 2026Normative Robustness as a Frontier for Non-Verifiable Reasoning in LLMscited by 0

50 most cited of 58 All 58 (JSON)