# Multilingual != Multicultural: Evaluating Gaps Between Multilingual Capabilities and Cultural Alignment in LLMs

arXiv: https://arxiv.org/abs/2502.16534 · HTML: https://mlsanity.com/paper/2502.16534/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 76 references that are in the map.
- First posted on arXiv 23 Feb 2025.

## Builds on (most cited first, 30 of 37)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset](https://mlsanity.com/paper/2309.11998/) (2023)
- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://mlsanity.com/paper/2402.07827/) (2024)
- [The Impact of AI on Developer Productivity: Evidence from GitHub Copilot](https://mlsanity.com/paper/2302.06590/) (2023)
- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://mlsanity.com/paper/2103.12028/) (2021)
- [Predictability and Surprise in Large Generative Models](https://mlsanity.com/paper/2202.07785/) (2022)
- [Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study](https://mlsanity.com/paper/2303.17466/) (2023)
- [The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation](https://mlsanity.com/paper/2301.01768/) (2023)
- [Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models](https://mlsanity.com/paper/2402.16786/) (2024)
- [MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering](https://mlsanity.com/paper/2007.15207/) (2020)
- [Probing Pre-Trained Language Models for Cross-Cultural Differences in Values](https://mlsanity.com/paper/2203.13722/) (2022)
- [A Review of Sparse Expert Models in Deep Learning](https://mlsanity.com/paper/2209.01667/) (2022)
- [Investigating Cultural Alignment of Large Language Models](https://mlsanity.com/paper/2402.13231/) (2024)
- [Auditing large language models: a three-layered approach](https://mlsanity.com/paper/2302.08500/) (2023)
- [Cultural Alignment in Large Language Models: An Explanatory Analysis Based on Hofstede's Cultural Dimensions](https://mlsanity.com/paper/2309.12342/) (2023)
- [The Ghost in the Machine has an American accent: value conflict in GPT-3](https://mlsanity.com/paper/2203.07785/) (2022)
- [Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Models](https://mlsanity.com/paper/2204.08110/) (2022)
- [Multilingual Language Models are not Multicultural: A Case Study in Emotion](https://mlsanity.com/paper/2307.01370/) (2023)

## Built on by (most cited first, 30 of 39)

- [Crosslingual Reasoning through Test-Time Scaling](https://mlsanity.com/paper/2505.05408/) (2025)
- [Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Dataset](https://mlsanity.com/paper/2507.09650/) (2025)
- [Language-Specific Latent Process Hinders Cross-Lingual Performance](https://mlsanity.com/paper/2505.13141/) (2025)
- [CARE: Multilingual Human Preference Learning for Cultural Awareness](https://mlsanity.com/paper/2504.05154/) (2025)
- [NileChat: Towards Linguistically Diverse and Culturally Aware LLMs for Local Communities](https://mlsanity.com/paper/2505.18383/) (2025)
- [DRISHTIKON: A Multimodal Multilingual Benchmark for Testing Language Models' Understanding on Indian Culture](https://mlsanity.com/paper/2509.19274/) (2025)
- [Meaning Is Not A Metric: Using LLMs to make cultural context legible at scale](https://mlsanity.com/paper/2505.23785/) (2025)
- [The Hidden Space of Safety: Understanding Preference-Tuned LLMs in Multilingual context](https://mlsanity.com/paper/2504.02708/) (2025)
- [CAReDiO: Cultural Alignment via Representativeness and Distinctiveness Guided Data Optimization](https://mlsanity.com/paper/2504.08820/) (2025)
- [Do Large Language Models Truly Understand Cross-cultural Differences?](https://mlsanity.com/paper/2512.07075/) (2025)
- [Multilingual Amnesia: On the Transferability of Unlearning in Multilingual LLMs](https://mlsanity.com/paper/2601.05641/) (2026)
- [Language Specific Knowledge: Do Models Know Better in X than in English?](https://mlsanity.com/paper/2505.14990/) (2025)
- [Beg to Differ: Understanding Reasoning-Answer Misalignment Across Languages](https://mlsanity.com/paper/2512.22712/) (2025)
- [Mind the Gap in Cultural Alignment: Task-Aware Culture Management for Large Language Models](https://mlsanity.com/paper/2602.22475/) (2026)
- [The Culture Funnel: You Can't Align What isn't in the Data](https://mlsanity.com/paper/2606.13808/) (2026)
- [Research Superalignment Should Advance Now with Alternating Competence and Conformity Optimization](https://mlsanity.com/paper/2503.07660/) (2025)
- [MateInfoUB: A Real-World Benchmark for Testing LLMs in Competitive, Multilingual, and Multimodal Educational Tasks](https://mlsanity.com/paper/2507.03162/) (2025)
- [Language Models Entangle Language and Culture](https://mlsanity.com/paper/2601.15337/) (2026)
- [Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebook](https://mlsanity.com/paper/2604.06210/) (2026)
- [Expressing Social Emotions: Misalignment Between LLMs and Human Cultural Emotion Norms](https://mlsanity.com/paper/2604.16757/) (2026)
- [The GaoYao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Models](https://mlsanity.com/paper/2604.20225/) (2026)
- [MADE: Beyond Scoring via a Multilingual Agentic Diagnosing Engine for Fine-Grained Evaluation Insights](https://mlsanity.com/paper/2606.07020/) (2026)
- [Benchmarking Frontier LLMs on Arabic Cultural and Sociolinguistic Knowledge: A Cross-Evaluation Framework with Human SME Ground Truth](https://mlsanity.com/paper/2607.00139/) (2026)
- [MIDB: Multilingual Instruction Data Booster for Enhancing Cultural Equality in Multilingual Instruction Synthesis](https://mlsanity.com/paper/2505.17671/) (2025)
- [Misalignment of LLM-Generated Personas with Human Perceptions in Low-Resource Settings](https://mlsanity.com/paper/2512.02058/) (2025)
- [Macaron: Controlled, Human-Written Benchmark for Multilingual and Multicultural Reasoning via Template-Filling](https://mlsanity.com/paper/2602.10732/) (2026)
- [Designing for Disagreement: Front-End Guardrails for Assistance Allocation in LLM-Enabled Robots](https://mlsanity.com/paper/2603.16537/) (2026)
- [Toward Culturally Grounded Natural Language Processing](https://mlsanity.com/paper/2603.26013/) (2026)
- [Grounding Text Embeddings in Stakeholder Associations](https://mlsanity.com/paper/2605.27168/) (2026)
- [When English Rewrites Local Knowledge: Global Narrative Dominance in Large Language Models](https://mlsanity.com/paper/2605.30481/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are becoming increasingly capable across global languages. However, the ability to communicate across languages does not necessarily translate to appropriate cultural representations. A key concern is US-centric bias, where LLMs reflect US rather than local cultural values. We propose a novel methodology that compares LLM-generated response distributions against population-level opinion data from the World Value Survey across four languages (Danish, Dutch, English, and Portuguese). Using a rigorous linear mixed-effects regression framework, we compare two families of models: Google's Gemma models (2B--27B parameters) and successive iterations of OpenAI's turbo-series. Across the families of models, we find no consistent relationships between language capabilities and cultural alignment. While the Gemma models have a positive correlation between language capability and cultural alignment across languages, the OpenAI models do not. Importantly, we find that self-consistency is a stronger predictor of multicultural alignment than multilingual capabilities. Our results demonstrate that achieving meaningful cultural alignment requires dedicated effort beyond improving general language capabilities.