# Evaluating Cultural Awareness of LLMs for Yoruba, Malayalam, and English

arXiv: https://arxiv.org/abs/2410.01811 · HTML: https://mlsanity.com/paper/2410.01811/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 50 references that are in the map.
- First posted on arXiv 14 Sep 2024.

## Builds on (most cited first, 27 of 27)

- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [A Multiscale Visualization of Attention in the Transformer Model](https://mlsanity.com/paper/1906.05714/) (2019)
- [Data and its (dis)contents: A survey of dataset development and use in machine learning research](https://mlsanity.com/paper/2012.05345/) (2020)
- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://mlsanity.com/paper/2103.12028/) (2021)
- [BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages](https://mlsanity.com/paper/2406.09948/) (2024)
- [Having Beer after Prayer? Measuring Cultural Bias in Large Language Models](https://mlsanity.com/paper/2305.14456/) (2023)
- [Cultural Alignment in Large Language Models: An Explanatory Analysis Based on Hofstede's Cultural Dimensions](https://mlsanity.com/paper/2309.12342/) (2023)
- [Human-centered Explainable AI: Towards a Reflective Sociotechnical Approach](https://mlsanity.com/paper/2002.01092/) (2020)
- [The relationship between trust in AI and trustworthy machine learning technologies](https://mlsanity.com/paper/1912.00782/) (2019)
- [What Does Explainable AI Really Mean? A New Conceptualization of Perspectives](https://mlsanity.com/paper/1710.00794/) (2017)
- [A Bibliometric Review of Large Language Models Research from 2017 to 2023](https://mlsanity.com/paper/2304.02020/) (2023)
- [How Well Do LLMs Represent Values Across Cultures? Empirical Analysis of LLM Responses Based on Hofstede Cultural Dimensions](https://mlsanity.com/paper/2406.14805/) (2024)
- [LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedback](https://mlsanity.com/paper/2406.01771/) (2024)
- [NormAd: A Framework for Measuring the Cultural Adaptability of Large Language Models](https://mlsanity.com/paper/2404.12464/) (2024)
- [Comparing Biases and the Impact of Multilingual Training across Multiple Languages](https://mlsanity.com/paper/2305.11242/) (2023)
- [Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayings](https://mlsanity.com/paper/2309.08591/) (2023)
- [CDEval: A Benchmark for Measuring the Cultural Dimensions of Large Language Models](https://mlsanity.com/paper/2311.16421/) (2023)
- [Diversity and Inclusion in Artificial Intelligence](https://mlsanity.com/paper/2305.12728/) (2023)
- [A Measure for Transparent Comparison of Linguistic Diversity in Multilingual NLP Data Sets](https://mlsanity.com/paper/2403.03909/) (2024)
- [Black-box language model explanation by context length probing](https://mlsanity.com/paper/2212.14815/) (2022)
- [Beyond Metrics: Evaluating LLMs' Effectiveness in Culturally Nuanced, Low-Resource Real-World Scenarios](https://mlsanity.com/paper/2406.00343/) (2024)
- [Towards a Praxis for Intercultural Ethics in Explainable AI](https://mlsanity.com/paper/2304.11861/) (2023)

## Built on by (most cited first, 8 of 8)

- [Break the Checkbox: Challenging Closed-Style Evaluations of Cultural Alignment in LLMs](https://mlsanity.com/paper/2502.08045/) (2025)
- [Meta-Cultural Competence: Climbing the Right Hill of Cultural Awareness](https://mlsanity.com/paper/2502.09637/) (2025)
- [Multilingual Amnesia: On the Transferability of Unlearning in Multilingual LLMs](https://mlsanity.com/paper/2601.05641/) (2026)
- [Through the Prism of Culture: Evaluating LLMs' Understanding of Indian Subcultures and Traditions](https://mlsanity.com/paper/2501.16748/) (2025)
- [LAG-MMLU: Benchmarking Frontier LLM Understanding in Latvian and Giriama](https://mlsanity.com/paper/2503.11911/) (2025)
- [Amplify Initiative: Building A Localized Data Platform for Globalized AI](https://mlsanity.com/paper/2504.14105/) (2025)
- [Cultural Authenticity: Comparing LLM Cultural Representations to Native Human Expectations](https://mlsanity.com/paper/2604.03493/) (2026)
- [Designing LLMs for cultural sensitivity: Evidence from English-Japanese translation](https://mlsanity.com/paper/2509.11921/) (2025)

## Abstract (arXiv)

Although LLMs have been extremely effective in a large number of complex tasks, their understanding and functionality for regional languages and cultures are not well studied. In this paper, we explore the ability of various LLMs to comprehend the cultural aspects of two regional languages: Malayalam (state of Kerala, India) and Yoruba (West Africa). Using Hofstede's six cultural dimensions: Power Distance (PDI), Individualism (IDV), Motivation towards Achievement and Success (MAS), Uncertainty Avoidance (UAV), Long Term Orientation (LTO), and Indulgence (IVR), we quantify the cultural awareness of LLM-based responses. We demonstrate that although LLMs show a high cultural similarity for English, they fail to capture the cultural nuances across these 6 metrics for Malayalam and Yoruba. We also highlight the need for large-scale regional language LLM training with culturally enriched datasets. This will have huge implications for enhancing the user experience of chat-based LLMs and also improving the validity of large-scale LLM agent-based market research.