# On the Credibility of Evaluating LLMs using Survey Questions

arXiv: https://arxiv.org/abs/2602.04033 · HTML: https://mlsanity.com/paper/2602.04033/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 43 references that are in the map.
- First posted on arXiv 3 Feb 2026.

## Builds on (most cited first, 23 of 23)

- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models](https://mlsanity.com/paper/2305.08283/) (2023)
- [Evaluating the Moral Beliefs Encoded in LLMs](https://mlsanity.com/paper/2307.14324/) (2023)
- [Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study](https://mlsanity.com/paper/2303.17466/) (2023)
- [Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models](https://mlsanity.com/paper/2402.16786/) (2024)
- [Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation](https://mlsanity.com/paper/2005.10283/) (2020)
- [EuroLLM: Multilingual Language Models for Europe](https://mlsanity.com/paper/2409.16235/) (2024)
- [On Decoding Strategies for Neural Text Generators](https://mlsanity.com/paper/2203.15721/) (2022)
- [Are Large Language Models Consistent over Value-laden Questions?](https://mlsanity.com/paper/2407.02996/) (2024)
- [Beyond prompt brittleness: Evaluating the reliability and consistency of political worldviews in LLMs](https://mlsanity.com/paper/2402.17649/) (2024)
- [Assessing LLMs for Moral Value Pluralism](https://mlsanity.com/paper/2312.10075/) (2023)
- [An Evaluation of Cultural Value Alignment in LLM](https://mlsanity.com/paper/2504.08863/) (2025)
- [Are Large Language Models Moral Hypocrites? A Study Based on Moral Foundations](https://mlsanity.com/paper/2405.11100/) (2024)
- [Whose Morality Do They Speak? Unraveling Cultural Bias in Multilingual Language Models](https://mlsanity.com/paper/2412.18863/) (2024)
- [LLMs and Cultural Values: the Impact of Prompt Language and Explicit Cultural Framing](https://mlsanity.com/paper/2511.03980/) (2025)
- [Demonstrations of the Potential of AI-based Political Issue Polling](https://mlsanity.com/paper/2307.04781/) (2023)
- [Are Language Models More Like Libraries or Like Librarians? Bibliotechnism, the Novel Reference Problem, and the Attitudes of LLMs](https://mlsanity.com/paper/2401.04854/) (2024)
- [Decoding Multilingual Moral Preferences: Unveiling LLM's Biases Through the Moral Machine Experiment](https://mlsanity.com/paper/2407.15184/) (2024)
- [Cultural Fidelity in Large-Language Models: An Evaluation of Online Language Resources as a Driver of Model Performance in Value Representation](https://mlsanity.com/paper/2410.10489/) (2024)
- [Striking a Balance: Alleviating Inconsistency in Pre-trained Models for Symmetric Classification Tasks](https://mlsanity.com/paper/2203.13491/) (2022)
- [Multilingual Political Views of Large Language Models: Identification and Steering](https://mlsanity.com/paper/2507.22623/) (2025)
- [Exploring Large Language Models on Cross-Cultural Values in Connection with Training Methodology](https://mlsanity.com/paper/2412.08846/) (2024)

## Built on by (most cited first, 3 of 3)

- [The Shared Latent Structure of LLM Psychometric Responses](https://mlsanity.com/paper/2605.05080/) (2026)
- [Signal or Spurious Cue? A Randomized Audit of Survey-Country Metadata in LLM Social Inference](https://mlsanity.com/paper/2608.06085/) (2026)
- [How do LLMs Evaluate Perceived Moral Agency? Investigating Moral Decision-Making in Human-Artificial Agents Interactions](https://mlsanity.com/paper/2609.05037/) (2026)

## Abstract (arXiv)

Recent studies evaluate the value orientation of large language models (LLMs) using adapted social surveys, typically by prompting models with survey questions and comparing their responses to average human responses. This paper identifies limitations in this methodology that, depending on the exact setup, can lead to both underestimating and overestimating the similarity of value orientation. Using the World Value Survey in three languages across five countries, we demonstrate that prompting methods (direct vs. chain-of-thought) and decoding strategies (greedy vs. sampling) significantly affect results. To assess the interaction between answers, we introduce a novel metric, self-correlation distance. This metric measures whether LLMs maintain consistent relationships between answers across different questions, as humans do. This indicates that even a high average agreement with human data, when considering LLM responses independently, does not guarantee structural alignment in responses. Additionally, we reveal a weak correlation between two common evaluation metrics, mean-squared distance and KL divergence, which assume that survey answers are independent of each other. For future research, we recommend CoT prompting, sampling-based decoding with dozens of samples, and robust analysis using multiple metrics, including self-correlation distance.