mlsanityData to 14 Oct 2026

GYROval: A Robust Benchmark for Cultural Value Orientation in Large Language Models

arXiv 2609.18384 ↗First posted 16 Sep 2026cs.HC cs.AI cs.CYcited by 0

Not on the map. Why?

Abstract (arXiv)

We present a robust benchmark for measuring cultural value orientation in large language models on the two Inglehart-Welzel axes over several domains and roles (hence GYROval - Gridded Yielding of Robust value Orientation), together with the results of administering it to twenty models. Items are binary contrastive scenarios in the sense introduced by CDEval: both options are legitimate courses of action, neither is correct, there is no answer key, and a model's score on an axis is the proportion of its responses falling on the counted pole. Eleven of the twenty models were additionally administered a paired Russian translation of the identical items and a second sampling temperature. The instrument is publicly released in both languages. Stability was assessed by treating the vignette as the unit of analysis, ranking the models within the levels of each perturbation factor, and summarising the agreement between levels by tie-corrected Kendall's W against an empirical permutation null.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 40 of its 65 references that are in the map. First posted on arXiv 16 Sep 2026.

Builds on

40 of 65 references.

References · 40

  1. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  2. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  3. 2020Aligning AI With Shared Human Valuescited by 723
  4. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  5. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 310
  6. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 263
  7. 2020Social Chemistry 101: Learning to Reason about Social and Moral Normscited by 255
  8. 2023Evaluating the Moral Beliefs Encoded in LLMscited by 215
  9. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 179
  10. 2024BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languagescited by 173
  11. 2023Cultural Bias and Cultural Alignment of Large Language Modelscited by 149
  12. 2024Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluationscited by 148
28 more
  1. 2021Can Machines Learn Morality? The Delphi Experimentcited by 141
  2. 2023Moral Foundations of Large Language Modelscited by 85
  3. 2024The Political Preferences of LLMscited by 84
  4. 2023Questioning the Survey Responses of Large Language Modelscited by 74
  5. 2024DailyDilemmas: Revealing Value Preferences of LLMs with Quandaries of Daily Lifecited by 59
  6. 2024Large Language Models Reflect the Ideology of their Creatorscited by 57
  7. 2023Large Language Models as Superpositions of Cultural Perspectivescited by 49
  8. 2024IrokoBench: A New Benchmark for African Languages in the Age of Large Language Modelscited by 43
  9. 2026Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thoughtcited by 40
  10. 2025Break the Checkbox: Challenging Closed-Style Evaluations of Cultural Alignment in LLMscited by 26
  11. 2023CDEval: A Benchmark for Measuring the Cultural Dimensions of Large Language Modelscited by 23
  12. 2025Value Portrait: Assessing Language Models' Values through Psychometrically and Ecologically Valid Itemscited by 18
  13. 2025Will AI Tell Lies to Save Sick Children? Litmus-Testing AI Values Prioritization with AIRiskDilemmascited by 18
  14. 2024Stick to your Role! Stability of Personal Values Expressed in Large Language Modelscited by 14
  15. 2025Prompt Perturbations Reveal Human-Like Biases in Large Language Model Survey Responsescited by 13
  16. 2025Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Culturescited by 10
  17. 2025From Prompts to Constructs: A Dual-Validity Framework for Large Language Model Research in Psychologycited by 9
  18. 2025Revisiting LLM Value Probing Strategies: Are They Robust and Expressive?cited by 8
  19. 2026Deployment Decision Reliability: A Generalizability-Theory Framework for Sizing Long-Horizon Agent Evaluationscited by 4
  20. 2025CCD-Bench: Probing Cultural Conflict in Large Language Model Decision-Makingcited by 3
  21. 2025The Alignment Veto: How Safety Training Suppresses Cultural Knowledge in LLMscited by 3
  22. 2026Cultural Value Alignment Via Latent Activation Steering in Large Language Modelscited by 3
  23. 2026Scenario-based Probing and Steering Cultural Values in Large Language Models--Extended Versioncited by 2
  24. 2026LLMs Contain Multitudes: How Deployment Context Reshapes Model-Level Preferences and Valuescited by 2
  25. 2026Apparent Psychological Profiles of Large Language Models are Largely a Measurement Artifactcited by 2
  26. 2026Beyond Value Benchmarks: Measuring Value-Structure Alignment in Large Language Models via Symmetric Q-Sortscited by 1
  27. 2026STONIC: A Layered Measurement Contract for LLM Value Profilingcited by 1
  28. 2026Measurement Validity in LLM Cultural Alignmentcited by 1

Built on by

Not cited yet.