mlsanityData to 7 Oct 2026

Break the Checkbox: Challenging Closed-Style Evaluations of Cultural Alignment in LLMs

arXiv 2502.08045 ↗First posted 12 Feb 2025cs.CL cs.AI cs.CYcited by 26

Not on the map. Why?

Abstract (arXiv)

A large number of studies rely on closed-style multiple-choice surveys to evaluate cultural alignment in Large Language Models (LLMs). In this work, we challenge this constrained evaluation paradigm and explore more realistic, unconstrained approaches. Using the World Values Survey (WVS) and Hofstede Cultural Dimensions as case studies, we demonstrate that LLMs exhibit stronger cultural alignment in less constrained settings, where responses are not forced. Additionally, we show that even minor changes, such as reordering survey choices, lead to inconsistent outputs, exposing the limitations of closed-style evaluations. Our findings advocate for more robust and flexible evaluation frameworks that focus on specific cultural proxies, encouraging more nuanced and accurate assessments of cultural alignment in LLMs.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 22 of its 64 references that are in the map. First posted on arXiv 12 Feb 2025.

Builds on

22 of 64 references.

References · 22

  1. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  2. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  3. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
  4. 2022Challenges and Strategies in Cross-Cultural NLPcited by 216
  5. 2023Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Studycited by 203
  6. 2024Towards Measuring and Modeling "Culture" in LLMs: A Surveycited by 166
  7. 2024Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Modelscited by 162
  8. 2022Probing Pre-Trained Language Models for Cross-Cultural Differences in Valuescited by 147
  9. 2023Cultural Alignment in Large Language Models: An Explanatory Analysis Based on Hofstede's Cultural Dimensionscited by 111
  10. 2022The Ghost in the Machine has an American accent: value conflict in GPT-3cited by 105
  11. 2024WorldValuesBench: A Large-Scale Benchmark Dataset for Multi-Cultural Value Awareness of Language Modelscited by 59
  12. 2024DailyDilemmas: Revealing Value Preferences of LLMs with Quandaries of Daily Lifecited by 57
10 more
  1. 2023Multilingual Language Models are not Multicultural: A Case Study in Emotioncited by 54
  2. 2024How Well Do LLMs Represent Values Across Cultures? Empirical Analysis of LLM Responses Based on Hofstede Cultural Dimensionscited by 51
  3. 2022Theory-Grounded Measurement of U.S. Social Stereotypes in English Language Modelscited by 35
  4. 2023Assessing LLMs for Moral Value Pluralismcited by 34
  5. 2023EtiCor: Corpus for Analyzing LLMs for Etiquettescited by 33
  6. 2024Self-Alignment: Improving Alignment of Cultural Values in LLMs via In-Context Learningcited by 18
  7. 2024Evaluating Cultural Awareness of LLMs for Yoruba, Malayalam, and Englishcited by 8
  8. 2024Cultural Fidelity in Large-Language Models: An Evaluation of Online Language Resources as a Driver of Model Performance in Value Representationcited by 7
  9. 2024Large Language Models as Mirrors of Societal Moral Standardscited by 5
  10. 2024LLMs as mirrors of societal moral standards: reflection of cultural divergence and agreement across ethical topicscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 22.

26 papers.

Sort:

By area:

  1. 2026Tears or Cheers? Benchmarking LLMs via Culturally Elicited Distinct Affective Responsescited by 6
  2. 2025EVALUESTEER: Measuring Reward Model Steerability Towards Values and Preferencescited by 3
  3. 2026Culturally Grounded Personas in Large Language Models: Characterization and Alignment with Socio-Psychological Value Frameworkscited by 3
  4. 2026Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMscited by 3
  5. 2026Measuring Opinion Bias and Sycophancy via LLM-based Persuasioncited by 3
  6. 2026Cultural Value Alignment Via Latent Activation Steering in Large Language Modelscited by 3
  7. 2026Every Token Counts: Exact Likert-Scale Distributions for Measuring LLM Attitudes and Biasescited by 3
  8. 2025A Game-Theoretic Negotiation Framework for Cross-Cultural Consensus in LLMscited by 2
  9. 2025Semantic Label Drift in Cross-Cultural Translationcited by 2
  10. 2026Cultural Compass: A Framework for Organizing Societal Norms to Detect Violations in Human-AI Conversationscited by 2
  11. 2026Leveraging Wikidata for Geographically Informed Sociocultural Bias Dataset Creation: Application to Latin Americacited by 2
  12. 2026Scenario-based Probing and Steering Cultural Values in Large Language Models--Extended Versioncited by 2
14 more
  1. 2025From n-gram to Attention: How Model Architectures Learn and Propagate Bias in Language Modelingcited by 1
  2. 2026Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebookcited by 1
  3. 2026Towards Cross-lingual Values Judgment: A Consensus-Pluralism Perspectivecited by 0
  4. 2026Toward Culturally Grounded Natural Language Processingcited by 0
  5. 2026Culture-Aware Humorous Captioning: Multimodal Humor Generation across Cultural Contextscited by 0
  6. 2026Modeling Community Attitude through Reaction Tone: A Human-AI Collaborative Framework for Evaluating LLM Alignment with Linguistic Behaviors in Online Communitiescited by 0
  7. 2026Not What, But How: A Framework for Auditing LLM Responses across Positioning, Generalization, Anthropomorphism, and Maximscited by 0
  8. 2026LLMs Infer Cultural Context but Fail to Apply It When Respondingcited by 0
  9. 2026When Trivia Is Not Trivial: Everyday Knowledge Failures in Multilingual LLMscited by 0
  10. 2026Do LLM Debates Repeat Arguments Differently Across Languages?cited by 0
  11. 2026Computational Orientalism: Measuring Structural Discourse Bias in Large Language Models Using the Middle East Cultural Sensitivity Score (MECSS)cited by 0
  12. 2026GYROval: A Robust Benchmark for Cultural Value Orientation in Large Language Modelscited by 0
  13. 2026CultureMINE: Datasets and Methods for Improving the Cultural Capabilities of NLP Systemscited by 0
  14. 2026Supporting and Performing Culture from the Insidecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Randomness, Not Representation: The Unreliability of Evaluating Cultural Alignment in LLMs
  2. 2025Multilingual != Multicultural: Evaluating Gaps Between Multilingual Capabilities and Cultural Alignment in LLMs
  3. 2023CDEval: A Benchmark for Measuring the Cultural Dimensions of Large Language Models
  4. 2026Aligned but Flattened: Analyzing the Trade-off between Cultural Alignment and Diversity in LLMs
  5. 2026AlignCultura: Towards Culturally Aligned Large Language Models?
  6. 2026Progressing beyond Art Masterpieces or Touristic Clich'es: how to assess your LLMs for cultural alignment?
  7. 2025An Evaluation of Cultural Value Alignment in LLM
  8. 2026Understanding Cultural Alignment in Multilingual LLMs via Natural Debate Statements ↗