mlsanityData to 7 Oct 2026

On the Credibility of Evaluating LLMs using Survey Questions

arXiv 2602.04033 ↗First posted 3 Feb 2026cs.CL cs.AI cs.CYcited by 3

Not on the map. Why?

Abstract (arXiv)

Recent studies evaluate the value orientation of large language models (LLMs) using adapted social surveys, typically by prompting models with survey questions and comparing their responses to average human responses. This paper identifies limitations in this methodology that, depending on the exact setup, can lead to both underestimating and overestimating the similarity of value orientation. Using the World Value Survey in three languages across five countries, we demonstrate that prompting methods (direct vs. chain-of-thought) and decoding strategies (greedy vs. sampling) significantly affect results. To assess the interaction between answers, we introduce a novel metric, self-correlation distance. This metric measures whether LLMs maintain consistent relationships between answers across different questions, as humans do. This indicates that even a high average agreement with human data, when considering LLM responses independently, does not guarantee structural alignment in responses. Additionally, we reveal a weak correlation between two common evaluation metrics, mean-squared distance and KL divergence, which assume that survey answers are independent of each other. For future research, we recommend CoT prompting, sampling-based decoding with dozens of samples, and robust analysis using multiple metrics, including self-correlation distance.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 23 of its 43 references that are in the map. First posted on arXiv 3 Feb 2026.

Builds on

23 of 43 references.

References · 23

  1. 2023Whose Opinions Do Language Models Reflect?cited by 759
  2. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  3. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 257
  4. 2023Evaluating the Moral Beliefs Encoded in LLMscited by 211
  5. 2023Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Studycited by 203
  6. 2024Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Modelscited by 162
  7. 2020Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translationcited by 137
  8. 2024EuroLLM: Multilingual Language Models for Europecited by 103
  9. 2022On Decoding Strategies for Neural Text Generatorscited by 89
  10. 2024Are Large Language Models Consistent over Value-laden Questions?cited by 58
  11. 2024Beyond prompt brittleness: Evaluating the reliability and consistency of political worldviews in LLMscited by 55
  12. 2023Assessing LLMs for Moral Value Pluralismcited by 34
11 more
  1. 2025An Evaluation of Cultural Value Alignment in LLMcited by 28
  2. 2024Are Large Language Models Moral Hypocrites? A Study Based on Moral Foundationscited by 19
  3. 2024Whose Morality Do They Speak? Unraveling Cultural Bias in Multilingual Language Modelscited by 17
  4. 2025LLMs and Cultural Values: the Impact of Prompt Language and Explicit Cultural Framingcited by 17
  5. 2023Demonstrations of the Potential of AI-based Political Issue Pollingcited by 14
  6. 2024Are Language Models More Like Libraries or Like Librarians? Bibliotechnism, the Novel Reference Problem, and the Attitudes of LLMscited by 11
  7. 2024Decoding Multilingual Moral Preferences: Unveiling LLM's Biases Through the Moral Machine Experimentcited by 11
  8. 2024Cultural Fidelity in Large-Language Models: An Evaluation of Online Language Resources as a Driver of Model Performance in Value Representationcited by 7
  9. 2022Striking a Balance: Alleviating Inconsistency in Pre-trained Models for Symmetric Classification Taskscited by 6
  10. 2025Multilingual Political Views of Large Language Models: Identification and Steeringcited by 5
  11. 2024Exploring Large Language Models on Cross-Cultural Values in Connection with Training Methodologycited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026The Pinocchio Dimension: Phenomenality of Experience as the Primary Axis of LLM Psychometric Differencescited by 0
  2. 2026Signal or Spurious Cue? A Randomized Audit of Survey-Country Metadata in LLM Social Inferencecited by 0
  3. 2026How do LLMs Evaluate Perceived Moral Agency? Investigating Moral Decision-Making in Human-Artificial Agents Interactionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Measurement Validity in LLM Cultural Alignment
  2. 2025Survey-to-Behavior: Downstream Alignment of Human Values in LLMs via Survey Questions
  3. 2025On the Alignment of Large Language Models with Global Human Opinion
  4. 2025An Analysis of Large Language Models for Simulating User Responses in Surveys
  5. 2025Do LLMs Align Human Values Regarding Social Biases? Judging and Explaining Social Biases with LLMs
  6. 2026Polyglots or Multitudes? Multilingual LLM Answers to Value-laden Multiple-Choice Questions
  7. 2025Do Language Models Think Consistently? A Study of Value Preferences Across Varying Response Lengths
  8. 2023Do LLMs exhibit human-like response biases? A case study in survey design