mlsanityData to 14 Oct 2026

Evaluating Cultural Awareness of LLMs for Yoruba, Malayalam, and English

arXiv 2410.01811 ↗First posted 14 Sep 2024cs.CY cs.AI cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

Although LLMs have been extremely effective in a large number of complex tasks, their understanding and functionality for regional languages and cultures are not well studied. In this paper, we explore the ability of various LLMs to comprehend the cultural aspects of two regional languages: Malayalam (state of Kerala, India) and Yoruba (West Africa). Using Hofstede's six cultural dimensions: Power Distance (PDI), Individualism (IDV), Motivation towards Achievement and Success (MAS), Uncertainty Avoidance (UAV), Long Term Orientation (LTO), and Indulgence (IVR), we quantify the cultural awareness of LLM-based responses. We demonstrate that although LLMs show a high cultural similarity for English, they fail to capture the cultural nuances across these 6 metrics for Malayalam and Yoruba. We also highlight the need for large-scale regional language LLM training with culturally enriched datasets. This will have huge implications for enhancing the user experience of chat-based LLMs and also improving the validity of large-scale LLM agent-based market research.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 27 of its 50 references that are in the map. First posted on arXiv 14 Sep 2024.

Builds on

27 of 50 references.

References · 27

  1. 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,552
  2. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
  3. 2018A Survey Of Methods For Explaining Black Box Modelscited by 973
  4. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  5. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  6. 2023A Comprehensive Overview of Large Language Modelscited by 648
  7. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  8. 2020Data and its (dis)contents: A survey of dataset development and use in machine learning researchcited by 260
  9. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 243
  10. 2024BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languagescited by 173
  11. 2023Having Beer after Prayer? Measuring Cultural Bias in Large Language Modelscited by 153
  12. 2023Cultural Alignment in Large Language Models: An Explanatory Analysis Based on Hofstede's Cultural Dimensionscited by 112
15 more
  1. 2020Human-centered Explainable AI: Towards a Reflective Sociotechnical Approachcited by 90
  2. 2019The relationship between trust in AI and trustworthy machine learning technologiescited by 78
  3. 2017What Does Explainable AI Really Mean? A New Conceptualization of Perspectivescited by 72
  4. 2023A Bibliometric Review of Large Language Models Research from 2017 to 2023cited by 57
  5. 2024How Well Do LLMs Represent Values Across Cultures? Empirical Analysis of LLM Responses Based on Hofstede Cultural Dimensionscited by 52
  6. 2024LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedbackcited by 51
  7. 2024NormAd: A Framework for Measuring the Cultural Adaptability of Large Language Modelscited by 48
  8. 2023Comparing Biases and the Impact of Multilingual Training across Multiple Languagescited by 36
  9. 2023Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayingscited by 32
  10. 2023CDEval: A Benchmark for Measuring the Cultural Dimensions of Large Language Modelscited by 23
  11. 2023Diversity and Inclusion in Artificial Intelligencecited by 14
  12. 2024A Measure for Transparent Comparison of Linguistic Diversity in Multilingual NLP Data Setscited by 10
  13. 2022Black-box language model explanation by context length probingcited by 7
  14. 2024Beyond Metrics: Evaluating LLMs' Effectiveness in Culturally Nuanced, Low-Resource Real-World Scenarioscited by 7
  15. 2023Towards a Praxis for Intercultural Ethics in Explainable AIcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 2.

8 papers.

  1. 2025Break the Checkbox: Challenging Closed-Style Evaluations of Cultural Alignment in LLMscited by 26
  2. 2025Meta-Cultural Competence: Climbing the Right Hill of Cultural Awarenesscited by 8
  3. 2026Multilingual Amnesia: On the Transferability of Unlearning in Multilingual LLMscited by 6
  4. 2025Through the Prism of Culture: Evaluating LLMs' Understanding of Indian Subcultures and Traditionscited by 4
  5. 2025LAG-MMLU: Benchmarking Frontier LLM Understanding in Latvian and Giriamacited by 2
  6. 2025Amplify Initiative: Building A Localized Data Platform for Globalized AIcited by 1
  7. 2026Cultural Authenticity: Comparing LLM Cultural Representations to Native Human Expectationscited by 1
  8. 2025Designing LLMs for cultural sensitivity: Evidence from English-Japanese translationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Survey of Cultural Awareness in Language Models: Text and Beyond
  2. 2025Fluent but Foreign: Even Regional LLMs Lack Cultural Alignment
  3. 2025An Evaluation of Cultural Value Alignment in LLM
  4. 2025Towards Geo-Culturally Grounded LLM Generations
  5. 2024Navigating the Cultural Kaleidoscope: A Hitchhiker's Guide to Sensitivity in Large Language Models
  6. 2025CURE: Cultural Understanding and Reasoning Evaluation - A Framework for "Thick" Culture Alignment Evaluation in LLMs
  7. 2024Evaluating Cultural and Social Awareness of LLM Web Agents
  8. 2024CulturalTeaming: AI-Assisted Interactive Red-Teaming for Challenging LLMs' (Lack of) Multicultural Knowledge