mlsanityData to 7 Oct 2026

NormAd: A Framework for Measuring the Cultural Adaptability of Large Language Models

arXiv 2404.12464 ↗First posted 18 Apr 2024cs.CLcited by 47

Not on the map. Why?

Abstract (arXiv)

To be effectively and safely deployed to global user populations, large language models (LLMs) may need to adapt outputs to user values and cultures, not just know about them. We introduce NormAd, an evaluation framework to assess LLMs' cultural adaptability, specifically measuring their ability to judge social acceptability across varying levels of cultural norm specificity, from abstract values to explicit social norms. As an instantiation of our framework, we create NormAd-Eti, a benchmark of 2.6k situational descriptions representing social-etiquette related cultural norms from 75 countries. Through comprehensive experiments on NormAd-Eti, we find that LLMs struggle to accurately judge social acceptability across these varying degrees of cultural contexts and show stronger adaptability to English-centric cultures over those from the Global South. Even in the simplest setting where the relevant social norms are provided, the best LLMs' performance (< 82%) lags behind humans (> 95%). In settings with abstract values and country information, model performance drops substantially (< 60%), while human accuracy remains high (> 90%). Furthermore, we find that models are better at recognizing socially acceptable versus unacceptable situations. Our findings showcase the current pitfalls in socio-cultural reasoning of LLMs which hinder their adaptability for global audiences.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 27 of its 63 references that are in the map. First posted on arXiv 18 Apr 2024.

Builds on

27 of 63 references.

References · 27

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  7. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  8. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  9. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  10. 2020Social Chemistry 101: Learning to Reason about Social and Moral Normscited by 253
  11. 2022SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualizationcited by 166
  12. 2023Cultural Alignment in Large Language Models: An Explanatory Analysis Based on Hofstede's Cultural Dimensionscited by 111
15 more
  1. 2022ProsocialDialog: A Prosocial Backbone for Conversational Agentscited by 109
  2. 2022The Ghost in the Machine has an American accent: value conflict in GPT-3cited by 105
  3. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 86
  4. 2022Extracting Cultural Commonsense Knowledge at Scalecited by 76
  5. 2023SeeGULL: A Stereotype Benchmark with Broad Geo-Cultural Coverage Leveraging Generative Modelscited by 66
  6. 2024Unintended Impacts of LLM Alignment on Global Representationcited by 66
  7. 2023Ethical Reasoning over Moral Alignment: A Case and Framework for In-Context Ethical Policies in LLMscited by 64
  8. 2024Massively Multi-Cultural Knowledge Acquisition & LM Benchmarkingcited by 63
  9. 2022Re-contextualizing Fairness in NLP: The Case of Indiacited by 55
  10. 2024IndoCulture: Exploring Geographically-Influenced Cultural Commonsense Reasoning Across Eleven Indonesian Provincescited by 38
  11. 2023Assessing LLMs for Moral Value Pluralismcited by 34
  12. 2024CulturalTeaming: AI-Assisted Interactive Red-Teaming for Challenging LLMs' (Lack of) Multicultural Knowledgecited by 28
  13. 2024DOSA: A Dataset of Social Artifacts from Different Indian Geographical Subculturescited by 23
  14. 2024The Colorful Future of LLMs: Evaluating and Improving LLMs as Emotional Supporters for Queer Youthcited by 20
  15. 2024Designing for Harm Reduction: Communication Repair for Multicultural Users' Voice Interactionscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 17; 2026: 23.

47 papers.

Sort:

By area:

  1. 2025ELEPHANT: Measuring and understanding social sycophancy in LLMscited by 115
  2. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaborationcited by 111
  3. 2024Survey of Cultural Awareness in Language Models: Text and Beyondcited by 101
  4. 2024AI Suggestions Homogenize Writing Toward Western Styles and Diminish Cultural Nuancescited by 59
  5. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  6. 2025Epistemic Alignment: A Mediating Framework for User-LLM Knowledge Deliverycited by 14
  7. 2025Social World Modelscited by 14
  8. 2026Tiny Aya: Bridging Scale and Multilingual Depthcited by 14
  9. 2025BASIL: Bayesian Assessment of Sycophancy in LLMscited by 11
  10. 2025From Surveys to Narratives: Rethinking Cultural Value Adaptation in LLMscited by 9
  11. 2025Fluent but Foreign: Even Regional LLMs Lack Cultural Alignmentcited by 9
  12. 2024Evaluating Cultural Awareness of LLMs for Yoruba, Malayalam, and Englishcited by 8
35 more
  1. 2025CURE: Cultural Understanding and Reasoning Evaluation - A Framework for "Thick" Culture Alignment Evaluation in LLMscited by 6
  2. 2025Sycophantic AI Decreases Prosocial Intentions and Promotes Dependencecited by 5
  3. 2025Counterfactual Reasoning for Steerable Pluralistic Value Alignment of Large Language Modelscited by 5
  4. 2024KAHANI: Culturally-Nuanced Visual Storytelling Tool for Non-Western Culturescited by 4
  5. 2025Fair-PP: A Synthetic Dataset for Aligning LLM with Personalized Preferences of Social Equitycited by 4
  6. 2025Sycophancy Claims about Language Models: The Missing Human-in-the-Loopcited by 4
  7. 2025BenchHub: A Unified Benchmark Suite for Holistic and Customizable LLM Evaluationcited by 3
  8. 2025Zhyper: Factorized Hypernetworks for Conditioned LLM Fine-Tuningcited by 3
  9. 2026Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMscited by 3
  10. 2025Camellia: Benchmarking Cultural Biases in LLMs for Asian Languagescited by 2
  11. 2025Multilingual Agent-Based World Modeling for Social Sciencecited by 2
  12. 2026XCR-Bench: Benchmarking Cross-Cultural Reasoning in LLMs via Culture-Specific Items and Hall's Triadcited by 2
  13. 2026A Unified Framework to Quantify Cultural Intelligence of AIcited by 2
  14. 2026Cultural Perspectives and Expectations for Generative AI: A Global Survey Approachcited by 2
  15. 2026The Culture Funnel: You Can't Align What isn't in the Datacited by 2
  16. 2024Exploring Cross-lingual Latent Transplantation: Mutual Opportunities and Open Challengescited by 1
  17. 2025CRaFT: An Explanation-Based Framework for Evaluating Cultural Reasoning in Multilingual Language Modelscited by 1
  18. 2025BengaliMoralBench: A Benchmark for Auditing Moral Reasoning in Large Language Models within Bengali Language and Culturecited by 1
  19. 2026Cultural Authenticity: Comparing LLM Cultural Representations to Native Human Expectationscited by 1
  20. 2026CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Taskscited by 1
  21. 2026XL-SafetyBench: A Country-Grounded Cross-Cultural Benchmark for LLM Safety and Cultural Sensitivitycited by 1
  22. 2026Benchmarking Frontier LLMs on Arabic Cultural and Sociolinguistic Knowledge: A Cross-Evaluation Framework with Human SME Ground Truthcited by 1
  23. 2026MAPLE: Metadata Conditioned LLM Pretraining for Locale-Aware Question Answeringcited by 0
  24. 2026PPCR-IM: A System for Multi-layer DAG-based Public Policy Consequence Reasoning and Social Indicator Mappingcited by 0
  25. 2026Do LLMs Use Cultural Knowledge Without Being Told? A Multilingual Evaluation of Implicit Pragmatic Adaptationcited by 0
  26. 2026On the Cultural Anachronism and Temporal Reasoning in Vision Language Modelscited by 0
  27. 2026EUDAIMONIA: Evaluating Undesirable Dynamics in AIcited by 0
  28. 2026Characterizing Cultural Localization in AI-Generated Storiescited by 0
  29. 2026LLMs Infer Cultural Context but Fail to Apply It When Respondingcited by 0
  30. 2026MSQA: A Natively Sourced Multilingual and Multicultural SimpleQA Benchmarkcited by 0
  31. 2026CCBENCH: Assessing LLM Cultural Competence via Implicitly Signaled Norms using Health Queriescited by 0
  32. 2026Trusting sovereign language models as scientific instruments: evidence from Portugal's AMALIAcited by 0
  33. 2026Toward Cultural Alignment: Human-Centered Evaluation of Multimodal AI Stories Across Five African Communitiescited by 0
  34. 2026LLMs Mirror Country-Specific Gender Patterns If Asked, but Skew Male When Generating Media in Local Languagescited by 0
  35. 2026DiSCo: A Distribution-First Steering and Cultural Prior Evaluation Framework for Measuring Cultural Preference Bias in LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Whose Norms? Disentangling Cultural and Personal Alignment in Large Language Models
  2. 2023CDEval: A Benchmark for Measuring the Cultural Dimensions of Large Language Models
  3. 2026Aligned but Flattened: Analyzing the Trade-off between Cultural Alignment and Diversity in LLMs
  4. 2025Cultural Learning-Based Culture Adaptation of Language Models
  5. 2025Break the Checkbox: Challenging Closed-Style Evaluations of Cultural Alignment in LLMs
  6. 2025Randomness, Not Representation: The Unreliability of Evaluating Cultural Alignment in LLMs
  7. 2025Multilingual != Multicultural: Evaluating Gaps Between Multilingual Capabilities and Cultural Alignment in LLMs
  8. 2026Understanding Cultural Alignment in Multilingual LLMs via Natural Debate Statements ↗