mlsanityData to 7 Oct 2026

Towards Measuring and Modeling "Culture" in LLMs: A Survey

arXiv 2403.15412 ↗First posted 5 Mar 2024cs.CY cs.AI cs.CLcited by 166

Not on the map. Why?

Cited by 23 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present a survey of more than 90 recent papers that aim to study cultural representation and inclusion in large language models (LLMs). We observe that none of the studies explicitly define "culture, which is a complex, multifaceted concept; instead, they probe the models on some specially designed datasets which represent certain aspects of "culture". We call these aspects the proxies of culture, and organize them across two dimensions of demographic and semantic proxies. We also categorize the probing methods employed. Our analysis indicates that only certain aspects of ``culture,'' such as values and objectives, have been studied, leaving several other interesting and important facets, especially the multitude of semantic domains (Thompson et al., 2020) and aboutness (Hershcovich et al., 2022), unexplored. Two other crucial gaps are the lack of robustness of probing techniques and situated studies on the impact of cultural mis- and under-representation in LLM-based applications.

Key facts. Cited by 166 papers in the map (data through 7 Oct 2026). Builds on 67 of its 134 references that are in the map. First posted on arXiv 5 Mar 2024.

Builds on

67 of 134 references.

References · 67

  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  2. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  3. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  4. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  5. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  6. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  7. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  8. 2023Whose Opinions Do Language Models Reflect?cited by 759
  9. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  10. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  11. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  12. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 506
38 more
  1. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  2. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 258
  3. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 257
  4. 2020Social Chemistry 101: Learning to Reason about Social and Moral Normscited by 253
  5. 2022Challenges and Strategies in Cross-Cultural NLPcited by 216
  6. 2023Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Studycited by 203
  7. 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
  8. 2023Cultural Alignment in Large Language Models: An Explanatory Analysis Based on Hofstede's Cultural Dimensionscited by 111
  9. 2021Re-imagining Algorithmic Fairness in India and Beyondcited by 102
  10. 2023Aligning Language Models to User Opinionscited by 98
  11. 2023Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Dutiescited by 97
  12. 2023SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoningcited by 94
  13. 2023NLPositionality: Characterizing Design Biases of Datasets and Modelscited by 91
  14. 2023Pearl: Personalizing Large Language Model Writing Assistants with Generation-Calibrated Retrieverscited by 91
  15. 2023Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Promptingcited by 80
  16. 2022Extracting Cultural Commonsense Knowledge at Scalecited by 76
  17. 2023KoBBQ: Korean Bias Benchmark for Question Answeringcited by 71
  18. 2023SeeGULL: A Stereotype Benchmark with Broad Geo-Cultural Coverage Leveraging Generative Modelscited by 66
  19. 2023Ethical Reasoning over Moral Alignment: A Case and Framework for In-Context Ethical Policies in LLMscited by 64
  20. 2023NormBank: A Knowledge Bank of Situational Social Normscited by 60
  21. 2019How should my chatbot interact? A survey on human-chatbot interaction designcited by 56
  22. 2023Multilingual Language Models are not Multicultural: A Case Study in Emotioncited by 54
  23. 2023Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLUcited by 53
  24. 2023When Do Annotator Demographics Matter? Measuring the Influence of Annotator Demographics with the POPQUORN Datasetcited by 51
  25. 2022NormSAGE: Multi-Lingual Multi-Cultural Norm Discovery from Conversations On-the-Flycited by 48
  26. 2023Large Language Models as Superpositions of Cultural Perspectivescited by 48
  27. 2023Multi-lingual and Multi-cultural Figurative Language Understandingcited by 47
  28. 2023From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Modelscited by 45
  29. 2020RiSAWOZ: A Large-Scale Multi-Domain Wizard-of-Oz Dataset with Rich Semantic Annotations for Task-Oriented Dialogue Modelingcited by 40
  30. 2023Self-Assessment Tests are Unreliable Measures of LLM Personalitycited by 39
  31. 2023Are Personalized Stochastic Parrots More Dangerous? Evaluating Persona Biases in Dialogue Systemscited by 37
  32. 2024Can LLM Generate Culturally Relevant Commonsense QA Data? Case Study in Indonesian and Sundanesecited by 36
  33. 2023EtiCor: Corpus for Analyzing LLMs for Etiquettescited by 33
  34. 2024Gradient-Based Language Model Red Teamingcited by 33
  35. 2023Building Socio-culturally Inclusive Stereotype Resources with Community Engagementcited by 25
  36. 2023KoSBi: A Dataset for Mitigating Social Bias Risks Towards Safer Large Language Model Applicationcited by 24
  37. 2020Linguistic Profiling of a Neural Language Modelcited by 23
  38. 2021A Word on Machine Ethics: A Response to Jiang et al. (2021)cited by 23

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 30; 2025: 68; 2026: 68.

166 papers.

Sort:

By area:

  1. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmarkcited by 118
  2. 2024Survey of Cultural Awareness in Language Models: Text and Beyondcited by 101
  3. 2024Benchmarking Vision Language Models for Cultural Understandingcited by 100
  4. 2024AI Suggestions Homogenize Writing Toward Western Styles and Diminish Cultural Nuancescited by 59
  5. 2024Are Large Language Models Consistent over Value-laden Questions?cited by 58
  6. 2024CULTURE-GEN: Revealing Global Cultural Perception in Language Models through Natural Language Promptingcited by 55
  7. 2024All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languagescited by 55
  8. 2024Language Model Alignment in Multilingual Trolley Problemscited by 46
  9. 2024SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languagescited by 41
  10. 2025CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countriescited by 40
  11. 2025Culture is Not Trivia: Sociocultural Theory for Cultural NLPcited by 40
  12. 2024WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisinescited by 39
38 more
  1. 2024CaLMQA: Exploring culturally specific long-form question answering across 23 languagescited by 31
  2. 2024CulturalBench: A Robust, Diverse, and Challenging Cultural Benchmark by Human-AI CulturalTeamingcited by 27
  3. 2024LLM-GLOBE: A Benchmark Evaluating the Cultural Values Embedded in LLM Outputcited by 27
  4. 2024Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Artcited by 26
  5. 2025Break the Checkbox: Challenging Closed-Style Evaluations of Cultural Alignment in LLMscited by 26
  6. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  7. 2025PalmX 2025: The First Shared Task on Benchmarking LLMs on Arabic and Islamic Culturecited by 20
  8. 2024The Potential and Challenges of Evaluating Attitudes, Opinions, and Values in Large Language Modelscited by 19
  9. 2024Self-Alignment: Improving Alignment of Cultural Values in LLMs via In-Context Learningcited by 18
  10. 2025Jawaher: A Multidialectal Dataset of Arabic Proverbs for LLM Benchmarkingcited by 18
  11. 2025SEA-HELM: Southeast Asian Holistic Evaluation of Language Modelscited by 17
  12. 2025Cultural Learning-Based Culture Adaptation of Language Modelscited by 16
  13. 2025Palm: A Culturally Inclusive and Linguistically Diverse Dataset for Arabic LLMscited by 15
  14. 2025CARE: Multilingual Human Preference Learning for Cultural Awarenesscited by 15
  15. 2025Diffusion Models Through a Global Lens: Are They Culturally Inclusive?cited by 14
  16. 2025Beyond Words: Exploring Cultural Value Sensitivity in Multimodal Modelscited by 14
  17. 2025CULEMO: Cultural Lenses on Emotion -- Benchmarking LLMs for Cross-Cultural Emotion Understandingcited by 14
  18. 2025CulturalFrames: Assessing Cultural Expectation Alignment in Text-to-Image Models and Evaluation Metricscited by 14
  19. 2025How Deep Is Representational Bias in LLMs? The Cases of Caste and Religioncited by 14
  20. 2026Tiny Aya: Bridging Scale and Multilingual Depthcited by 14
  21. 2025DRISHTIKON: A Multimodal Multilingual Benchmark for Testing Language Models' Understanding on Indian Culturecited by 13
  22. 2025Hire Your Anthropologist! Rethinking Culture Benchmarks Through an Anthropological Lenscited by 13
  23. 2025Time Travel: A Comprehensive Benchmark to Evaluate LMMs on Historical and Cultural Artifactscited by 10
  24. 2025Meaning Is Not A Metric: Using LLMs to make cultural context legible at scalecited by 10
  25. 2025Culture is Everywhere: A Call for Intentionally Cultural Evaluationcited by 10
  26. 2025TALES: A Taxonomy and Analysis of Cultural Representations in LLM-generated Storiescited by 10
  27. 2024Does Cross-Cultural Alignment Change the Commonsense Morality of Language Models?cited by 9
  28. 2024The Power of Many: Multi-Agent Multimodal Models for Cultural Image Captioningcited by 9
  29. 2025Fluent but Foreign: Even Regional LLMs Lack Cultural Alignmentcited by 9
  30. 2024CRAFT: Extracting and Tuning Cultural Instructions from the Wildcited by 8
  31. 2025Meta-Cultural Competence: Climbing the Right Hill of Cultural Awarenesscited by 8
  32. 2025Provocations from the Humanities for Generative AI Researchcited by 8
  33. 2025SaudiCulture: A Benchmark for Evaluating Large Language Models Cultural Competence within Saudi Arabiacited by 8
  34. 2025Multimodal Cultural Safety: Evaluation Framework and Alignment Strategiescited by 8
  35. 2025When Cultures Meet: Multicultural Text-to-Image Generationcited by 7
  36. 2025Unequal Opportunities: Examining the Bias in Geographical Recommendations by Large Language Modelscited by 7
  37. 2025AI-generated stories favour stability over change: homogeneity and cultural stereotyping in narratives generated by gpt-4o-minicited by 7
  38. 2025Position: LLMs Can be Good Tutors in English Educationcited by 6

50 most cited of 166 All 166 (JSON)