# Break the Checkbox: Challenging Closed-Style Evaluations of Cultural Alignment in LLMs

arXiv: https://arxiv.org/abs/2502.08045 · HTML: https://mlsanity.com/paper/2502.08045/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 64 references that are in the map.
- First posted on arXiv 12 Feb 2025.

## Builds on (most cited first, 22 of 22)

- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions](https://mlsanity.com/paper/2308.11483/) (2023)
- [Challenges and Strategies in Cross-Cultural NLP](https://mlsanity.com/paper/2203.10020/) (2022)
- [Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study](https://mlsanity.com/paper/2303.17466/) (2023)
- [Towards Measuring and Modeling "Culture" in LLMs: A Survey](https://mlsanity.com/paper/2403.15412/) (2024)
- [Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models](https://mlsanity.com/paper/2402.16786/) (2024)
- [Probing Pre-Trained Language Models for Cross-Cultural Differences in Values](https://mlsanity.com/paper/2203.13722/) (2022)
- [Cultural Alignment in Large Language Models: An Explanatory Analysis Based on Hofstede's Cultural Dimensions](https://mlsanity.com/paper/2309.12342/) (2023)
- [The Ghost in the Machine has an American accent: value conflict in GPT-3](https://mlsanity.com/paper/2203.07785/) (2022)
- [WorldValuesBench: A Large-Scale Benchmark Dataset for Multi-Cultural Value Awareness of Language Models](https://mlsanity.com/paper/2404.16308/) (2024)
- [DailyDilemmas: Revealing Value Preferences of LLMs with Quandaries of Daily Life](https://mlsanity.com/paper/2410.02683/) (2024)
- [Multilingual Language Models are not Multicultural: A Case Study in Emotion](https://mlsanity.com/paper/2307.01370/) (2023)
- [How Well Do LLMs Represent Values Across Cultures? Empirical Analysis of LLM Responses Based on Hofstede Cultural Dimensions](https://mlsanity.com/paper/2406.14805/) (2024)
- [Theory-Grounded Measurement of U.S. Social Stereotypes in English Language Models](https://mlsanity.com/paper/2206.11684/) (2022)
- [Assessing LLMs for Moral Value Pluralism](https://mlsanity.com/paper/2312.10075/) (2023)
- [EtiCor: Corpus for Analyzing LLMs for Etiquettes](https://mlsanity.com/paper/2310.18974/) (2023)
- [Self-Alignment: Improving Alignment of Cultural Values in LLMs via In-Context Learning](https://mlsanity.com/paper/2408.16482/) (2024)
- [Evaluating Cultural Awareness of LLMs for Yoruba, Malayalam, and English](https://mlsanity.com/paper/2410.01811/) (2024)
- [Cultural Fidelity in Large-Language Models: An Evaluation of Online Language Resources as a Driver of Model Performance in Value Representation](https://mlsanity.com/paper/2410.10489/) (2024)
- [Large Language Models as Mirrors of Societal Moral Standards](https://mlsanity.com/paper/2412.00956/) (2024)
- [LLMs as mirrors of societal moral standards: reflection of cultural divergence and agreement across ethical topics](https://mlsanity.com/paper/2412.00962/) (2024)

## Built on by (most cited first, 26 of 26)

- [Tears or Cheers? Benchmarking LLMs via Culturally Elicited Distinct Affective Responses](https://mlsanity.com/paper/2601.13024/) (2026)
- [Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs](https://mlsanity.com/paper/2604.19292/) (2026)
- [EVALUESTEER: Measuring Reward Model Steerability Towards Values and Preferences](https://mlsanity.com/paper/2510.06370/) (2025)
- [Culturally Grounded Personas in Large Language Models: Characterization and Alignment with Socio-Psychological Value Frameworks](https://mlsanity.com/paper/2601.22396/) (2026)
- [Measuring Opinion Bias and Sycophancy via LLM-based Persuasion](https://mlsanity.com/paper/2604.21564/) (2026)
- [Cultural Value Alignment Via Latent Activation Steering in Large Language Models](https://mlsanity.com/paper/2605.26365/) (2026)
- [Every Token Counts: Exact Likert-Scale Distributions for Measuring LLM Attitudes and Biases](https://mlsanity.com/paper/2608.10503/) (2026)
- [A Game-Theoretic Negotiation Framework for Cross-Cultural Consensus in LLMs](https://mlsanity.com/paper/2506.13245/) (2025)
- [Semantic Label Drift in Cross-Cultural Translation](https://mlsanity.com/paper/2510.25967/) (2025)
- [Cultural Compass: A Framework for Organizing Societal Norms to Detect Violations in Human-AI Conversations](https://mlsanity.com/paper/2601.07973/) (2026)
- [Leveraging Wikidata for Geographically Informed Sociocultural Bias Dataset Creation: Application to Latin America](https://mlsanity.com/paper/2603.10001/) (2026)
- [Scenario-based Probing and Steering Cultural Values in Large Language Models--Extended Version](https://mlsanity.com/paper/2606.11399/) (2026)
- [From n-gram to Attention: How Model Architectures Learn and Propagate Bias in Language Modeling](https://mlsanity.com/paper/2505.12381/) (2025)
- [Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebook](https://mlsanity.com/paper/2604.06210/) (2026)
- [Towards Cross-lingual Values Judgment: A Consensus-Pluralism Perspective](https://mlsanity.com/paper/2602.17283/) (2026)
- [Toward Culturally Grounded Natural Language Processing](https://mlsanity.com/paper/2603.26013/) (2026)
- [Culture-Aware Humorous Captioning: Multimodal Humor Generation across Cultural Contexts](https://mlsanity.com/paper/2604.18091/) (2026)
- [Modeling Community Attitude through Reaction Tone: A Human-AI Collaborative Framework for Evaluating LLM Alignment with Linguistic Behaviors in Online Communities](https://mlsanity.com/paper/2605.27388/) (2026)
- [Not What, But How: A Framework for Auditing LLM Responses across Positioning, Generalization, Anthropomorphism, and Maxims](https://mlsanity.com/paper/2606.02493/) (2026)
- [LLMs Infer Cultural Context but Fail to Apply It When Responding](https://mlsanity.com/paper/2606.17688/) (2026)
- [When Trivia Is Not Trivial: Everyday Knowledge Failures in Multilingual LLMs](https://mlsanity.com/paper/2607.21445/) (2026)
- [Do LLM Debates Repeat Arguments Differently Across Languages?](https://mlsanity.com/paper/2607.23442/) (2026)
- [Computational Orientalism: Measuring Structural Discourse Bias in Large Language Models Using the Middle East Cultural Sensitivity Score (MECSS)](https://mlsanity.com/paper/2608.18100/) (2026)
- [GYROval: A Robust Benchmark for Cultural Value Orientation in Large Language Models](https://mlsanity.com/paper/2609.18384/) (2026)
- [CultureMINE: Datasets and Methods for Improving the Cultural Capabilities of NLP Systems](https://mlsanity.com/paper/2609.22494/) (2026)
- [Supporting and Performing Culture from the Inside](https://mlsanity.com/paper/2609.32281/) (2026)

## Abstract (arXiv)

A large number of studies rely on closed-style multiple-choice surveys to evaluate cultural alignment in Large Language Models (LLMs). In this work, we challenge this constrained evaluation paradigm and explore more realistic, unconstrained approaches. Using the World Values Survey (WVS) and Hofstede Cultural Dimensions as case studies, we demonstrate that LLMs exhibit stronger cultural alignment in less constrained settings, where responses are not forced. Additionally, we show that even minor changes, such as reordering survey choices, lead to inconsistent outputs, exposing the limitations of closed-style evaluations. Our findings advocate for more robust and flexible evaluation frameworks that focus on specific cultural proxies, encouraging more nuanced and accurate assessments of cultural alignment in LLMs.