mlsanityData to 14 Oct 2026

Multilingual != Multicultural: Evaluating Gaps Between Multilingual Capabilities and Cultural Alignment in LLMs

arXiv 2502.16534 ↗First posted 23 Feb 2025cs.CL cs.AI cs.CYcited by 39

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) are becoming increasingly capable across global languages. However, the ability to communicate across languages does not necessarily translate to appropriate cultural representations. A key concern is US-centric bias, where LLMs reflect US rather than local cultural values. We propose a novel methodology that compares LLM-generated response distributions against population-level opinion data from the World Value Survey across four languages (Danish, Dutch, English, and Portuguese). Using a rigorous linear mixed-effects regression framework, we compare two families of models: Google's Gemma models (2B--27B parameters) and successive iterations of OpenAI's turbo-series. Across the families of models, we find no consistent relationships between language capabilities and cultural alignment. While the Gemma models have a positive correlation between language capability and cultural alignment across languages, the OpenAI models do not. Importantly, we find that self-consistency is a stronger predictor of multicultural alignment than multilingual capabilities. Our results demonstrate that achieving meaningful cultural alignment requires dedicated effort beyond improving general language capabilities.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Builds on 37 of its 76 references that are in the map. First posted on arXiv 23 Feb 2025.

Builds on

37 of 76 references.

References · 37

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  10. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  11. 2023LIMA: Less Is More for Alignmentcited by 1,100
  12. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
25 more
  1. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  2. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  3. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  4. 2023The Impact of AI on Developer Productivity: Evidence from GitHub Copilotcited by 274
  5. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 243
  6. 2022Predictability and Surprise in Large Generative Modelscited by 210
  7. 2023Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Studycited by 206
  8. 2023The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientationcited by 189
  9. 2024Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Modelscited by 162
  10. 2020MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answeringcited by 153
  11. 2022Probing Pre-Trained Language Models for Cross-Cultural Differences in Valuescited by 149
  12. 2022A Review of Sparse Expert Models in Deep Learningcited by 148
  13. 2024Investigating Cultural Alignment of Large Language Modelscited by 143
  14. 2023Auditing large language models: a three-layered approachcited by 137
  15. 2023Cultural Alignment in Large Language Models: An Explanatory Analysis Based on Hofstede's Cultural Dimensionscited by 112
  16. 2022The Ghost in the Machine has an American accent: value conflict in GPT-3cited by 105
  17. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 92
  18. 2023Multilingual Language Models are not Multicultural: A Case Study in Emotioncited by 54
  19. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Valuescited by 54
  20. 2022Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Modelscited by 49
  21. 2023Specific versus General Principles for Constitutional AIcited by 47
  22. 2025IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistancecited by 26
  23. 2024GPT-4 vs. Human Translators: A Comprehensive Evaluation of Translation Quality Across Languages, Domains, and Expertise Levelscited by 21
  24. 2024Do Multilingual Large Language Models Mitigate Stereotype Bias?cited by 13
  25. 2017Passing the Brazilian OAB Exam: data preparation and some experimentscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 16; 2026: 23.

39 papers.

Sort:

By area:

  1. 2025Crosslingual Reasoning through Test-Time Scalingcited by 46
  2. 2025Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Datasetcited by 35
  3. 2025Language-Specific Latent Process Hinders Cross-Lingual Performancecited by 21
  4. 2025CARE: Multilingual Human Preference Learning for Cultural Awarenesscited by 15
  5. 2025NileChat: Towards Linguistically Diverse and Culturally Aware LLMs for Local Communitiescited by 13
  6. 2025DRISHTIKON: A Multimodal Multilingual Benchmark for Testing Language Models' Understanding on Indian Culturecited by 13
  7. 2025Meaning Is Not A Metric: Using LLMs to make cultural context legible at scalecited by 10
  8. 2025The Hidden Space of Safety: Understanding Preference-Tuned LLMs in Multilingual contextcited by 6
  9. 2025CAReDiO: Cultural Alignment via Representativeness and Distinctiveness Guided Data Optimizationcited by 6
  10. 2025Do Large Language Models Truly Understand Cross-cultural Differences?cited by 6
  11. 2026Multilingual Amnesia: On the Transferability of Unlearning in Multilingual LLMscited by 6
  12. 2025Language Specific Knowledge: Do Models Know Better in X than in English?cited by 4
27 more
  1. 2025Beg to Differ: Understanding Reasoning-Answer Misalignment Across Languagescited by 4
  2. 2026Mind the Gap in Cultural Alignment: Task-Aware Culture Management for Large Language Modelscited by 3
  3. 2026The Culture Funnel: You Can't Align What isn't in the Datacited by 2
  4. 2025Research Superalignment Should Advance Now with Alternating Competence and Conformity Optimizationcited by 1
  5. 2025MateInfoUB: A Real-World Benchmark for Testing LLMs in Competitive, Multilingual, and Multimodal Educational Taskscited by 1
  6. 2026Language Models Entangle Language and Culturecited by 1
  7. 2026Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebookcited by 1
  8. 2026Expressing Social Emotions: Misalignment Between LLMs and Human Cultural Emotion Normscited by 1
  9. 2026The GaoYao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Modelscited by 1
  10. 2026MADE: Beyond Scoring via a Multilingual Agentic Diagnosing Engine for Fine-Grained Evaluation Insightscited by 1
  11. 2026Benchmarking Frontier LLMs on Arabic Cultural and Sociolinguistic Knowledge: A Cross-Evaluation Framework with Human SME Ground Truthcited by 1
  12. 2025MIDB: Multilingual Instruction Data Booster for Enhancing Cultural Equality in Multilingual Instruction Synthesiscited by 0
  13. 2025Misalignment of LLM-Generated Personas with Human Perceptions in Low-Resource Settingscited by 0
  14. 2026Macaron: Controlled, Human-Written Benchmark for Multilingual and Multicultural Reasoning via Template-Fillingcited by 0
  15. 2026Designing for Disagreement: Front-End Guardrails for Assistance Allocation in LLM-Enabled Robotscited by 0
  16. 2026Toward Culturally Grounded Natural Language Processingcited by 0
  17. 2026Grounding Text Embeddings in Stakeholder Associationscited by 0
  18. 2026When English Rewrites Local Knowledge: Global Narrative Dominance in Large Language Modelscited by 0
  19. 2026Do Large Language Models Encode Institutional Experience? Evidence from Cross-Linguistic Moral Reasoning Under Ambiguitycited by 0
  20. 2026Which Institutional Frameworks Do Chatbots Assume? Auditing Jurisdictional Defaults in Multilingual LLMscited by 0
  21. 2026BenSyc: Benchmarking Conversational Sycophancy and Human Alignment in LLMs for Bengali Contextscited by 0
  22. 2026The Shibboleth Effect: Auditing the Cross-Lingual Distributional Skew of Large Language Modelscited by 0
  23. 2026The Heterogeneous Safety Impacts of Benign Multilingual Fine-Tuningcited by 0
  24. 2026SPLIT: Cross-Lingual Empathy and Cultural Grounding in English and Ukrainian LLM Responsescited by 0
  25. 2026CCBENCH: Assessing LLM Cultural Competence via Implicitly Signaled Norms using Health Queriescited by 0
  26. 2026ADAGE: A Language-Agnostic Pipeline for Analogical Reasoning Evaluationcited by 0
  27. 2026Computational Orientalism: Measuring Structural Discourse Bias in Large Language Models Using the Middle East Cultural Sensitivity Score (MECSS)cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Break the Checkbox: Challenging Closed-Style Evaluations of Cultural Alignment in LLMs
  2. 2026Aligned but Flattened: Analyzing the Trade-off between Cultural Alignment and Diversity in LLMs
  3. 2025Randomness, Not Representation: The Unreliability of Evaluating Cultural Alignment in LLMs
  4. 2023CDEval: A Benchmark for Measuring the Cultural Dimensions of Large Language Models
  5. 2026Mind the Gap: Pitfalls of LLM Alignment with Asian Public Opinion
  6. 2026Understanding Cultural Alignment in Multilingual LLMs via Natural Debate Statements ↗
  7. 2025On the Alignment of Large Language Models with Global Human Opinion
  8. 2026Cross-Lingual Consensus: Aligning Multilingual Cultural Knowledge via Multilingual Self-Consistency