mlsanityData to 7 Oct 2026

WorldValuesBench: A Large-Scale Benchmark Dataset for Multi-Cultural Value Awareness of Language Models

arXiv 2404.16308 ↗First posted 25 Apr 2024cs.CL cs.CYcited by 59

Not on the map. Why?

Abstract (arXiv)

The awareness of multi-cultural human values is critical to the ability of language models (LMs) to generate safe and personalized responses. However, this awareness of LMs has been insufficiently studied, since the computer science community lacks access to the large-scale real-world data about multi-cultural values. In this paper, we present WorldValuesBench, a globally diverse, large-scale benchmark dataset for the multi-cultural value prediction task, which requires a model to generate a rating response to a value question based on demographic contexts. Our dataset is derived from an influential social science project, World Values Survey (WVS), that has collected answers to hundreds of value questions (e.g., social, economic, ethical) from 94,728 participants worldwide. We have constructed more than 20 million examples of the type "(demographic attributes, value question) rightarrow answer" from the WVS responses. We perform a case study using our dataset and show that the task is challenging for strong open and closed-source models. On merely 11.1%, 25.0%, 72.2%, and 75.0% of the questions, Alpaca-7B, Vicuna-7B-v1.5, Mixtral-8x7B-Instruct-v0.1, and GPT-3.5 Turbo can respectively achieve <0.2 Wasserstein 1-distance from the human normalized answer distributions. WorldValuesBench opens up new research avenues in studying limitations and opportunities in multi-cultural value awareness of LMs.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Builds on 8 of its 21 references that are in the map. First posted on arXiv 25 Apr 2024.

Builds on

8 of 21 references.

References · 8

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Whose Opinions Do Language Models Reflect?cited by 759
  3. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  4. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  5. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  6. 2022Probing Pre-Trained Language Models for Cross-Cultural Differences in Valuescited by 147
  7. 2022The Ghost in the Machine has an American accent: value conflict in GPT-3cited by 105
  8. 2023Constructive Large Language Models Alignment with Diverse Feedbackcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 20; 2026: 34.

59 papers.

Sort:

By area:

  1. 2024Evaluating Cultural and Social Awareness of LLM Web Agentscited by 43
  2. 2025CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countriescited by 40
  3. 2025Culture is Not Trivia: Sociocultural Theory for Cultural NLPcited by 40
  4. 2025The Bitter Lesson Learned from 2,000+ Multilingual Benchmarkscited by 31
  5. 2024Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Artcited by 26
  6. 2025Break the Checkbox: Challenging Closed-Style Evaluations of Cultural Alignment in LLMscited by 26
  7. 2024CulturePark: Boosting Cross-cultural Understanding in Large Language Modelscited by 24
  8. 2025Cultural Learning-Based Culture Adaptation of Language Modelscited by 16
  9. 2025Beyond Words: Exploring Cultural Value Sensitivity in Multimodal Modelscited by 14
  10. 2024Do LLMs have Consistent Values?cited by 13
  11. 2025TALES: A Taxonomy and Analysis of Cultural Representations in LLM-generated Storiescited by 10
  12. 2024CVLUE: A New Benchmark Dataset for Chinese Vision-Language Understanding Evaluationcited by 8
38 more
  1. 2025Modeling Earth-Scale Human-Like Societies with One Billion Agentscited by 8
  2. 2025Value Compass Benchmarks: A Platform for Fundamental and Validated Evaluation of LLMs Valuescited by 6
  3. 2025We Politely Insist: Your LLM Must Learn the Persian Art of Taarofcited by 6
  4. 2025Fair-PP: A Synthetic Dataset for Aligning LLM with Personalized Preferences of Social Equitycited by 4
  5. 2025AdAEM: An Adaptively and Automated Extensible Measurement of LLMs' Value Differencecited by 4
  6. 2025Evaluating LLM Adaptation to Sociodemographic Factors: User Profile vs. Dialogue Historycited by 4
  7. 2025Exploring Cultural Variations in Moral Judgments with Large Language Modelscited by 3
  8. 2026Toward Culturally Aligned LLMs through Ontology-Guided Multi-Agent Reasoningcited by 3
  9. 2026Culturally Grounded Personas in Large Language Models: Characterization and Alignment with Socio-Psychological Value Frameworkscited by 3
  10. 2026When Synthetic Users Fail: A Cross-Domain Benchmark of LLM-Simulated Human Survey Responsescited by 3
  11. 2026Every Token Counts: Exact Likert-Scale Distributions for Measuring LLM Attitudes and Biasescited by 3
  12. 2026CALM: Culturally Self-Aware Language Modelscited by 2
  13. 2026XCR-Bench: Benchmarking Cross-Cultural Reasoning in LLMs via Culture-Specific Items and Hall's Triadcited by 2
  14. 2025Neuron-Level Analysis of Cultural Understanding in Large Language Modelscited by 1
  15. 2025CRaFT: An Explanation-Based Framework for Evaluating Cultural Reasoning in Multilingual Language Modelscited by 1
  16. 2026HAG: Hierarchical Demographic Tree-based Agent Generation for Topic-Adaptive Simulationcited by 1
  17. 2026No Shortcuts to Culture: Indonesian Multi-hop Question Answering for Complex Cultural Understandingcited by 1
  18. 2026Prompt Programming for Cultural Bias and Alignment of Large Language Modelscited by 1
  19. 2026Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebookcited by 1
  20. 2026ValueGround: Evaluating Culture-Conditioned Visual Value Grounding in MLLMscited by 1
  21. 2026LKValues: Aligning Large Language Models with Sri Lankan Societal Valuescited by 1
  22. 2026Personalization, Personas, and Forecasting in Value Alignmentcited by 1
  23. 2025NurValues: Real-World Nursing Values Evaluation for Large Language Models in Clinical Contextcited by 0
  24. 2025ALIGN: Word Association Learning for Cultural Alignment in Large Language Modelscited by 0
  25. 2025The Realignment Problem: When Right becomes Wrong in LLMscited by 0
  26. 2025Cross-cultural value alignment frameworks for responsible AI governance: Evidence from China-West comparative analysiscited by 0
  27. 2026CuMA: Aligning LLMs with Sparse Cultural Values via Demographic-Aware Mixture of Adapterscited by 0
  28. 2026The Role of Dataset Linguistic Structure in the Cultural Awareness of Large Language Modelscited by 0
  29. 2026SOMtime the World Ain't Fair: Violating Fairness Using Self-Organizing Mapscited by 0
  30. 2026BanglaSocialBench: A Benchmark for Evaluating Sociopragmatic and Cultural Alignment of LLMs in Bangladeshi Social Interactioncited by 0
  31. 2026Toward Culturally Grounded Natural Language Processingcited by 0
  32. 2026Beyond Facts: Benchmarking Distributional Reading Comprehension in Large Language Modelscited by 0
  33. 2026Cross-Cultural Value Attribution in Large Vision-Language Modelscited by 0
  34. 2026Adaptive Querying with AI Persona Priorscited by 0
  35. 2026BenCSSmark: Making the Social Sciences Count in LLM Researchcited by 0
  36. 2026Not What, But How: A Framework for Auditing LLM Responses across Positioning, Generalization, Anthropomorphism, and Maximscited by 0
  37. 2026Occupational Prompting Reveals Cultural Bias in Large Language Modelscited by 0
  38. 2026LLMs Infer Cultural Context but Fail to Apply It When Respondingcited by 0

50 most cited of 59 All 59 (JSON)