# Cultural Fidelity in Large-Language Models: An Evaluation of Online Language Resources as a Driver of Model Performance in Value Representation

arXiv: https://arxiv.org/abs/2410.10489 · HTML: https://mlsanity.com/paper/2410.10489/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 71 references that are in the map.
- First posted on arXiv 14 Oct 2024.

## Builds on (most cited first, 17 of 17)

- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study](https://mlsanity.com/paper/2303.17466/) (2023)
- [Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models](https://mlsanity.com/paper/2402.16786/) (2024)
- [Probing Pre-Trained Language Models for Cross-Cultural Differences in Values](https://mlsanity.com/paper/2203.13722/) (2022)
- [A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translation](https://mlsanity.com/paper/2205.02022/) (2022)
- [Low-resource Languages: A Review of Past Work and Future Challenges](https://mlsanity.com/paper/2006.07264/) (2020)
- [How Well Do LLMs Represent Values Across Cultures? Empirical Analysis of LLM Responses Based on Hofstede Cultural Dimensions](https://mlsanity.com/paper/2406.14805/) (2024)
- [Assessing LLMs for Moral Value Pluralism](https://mlsanity.com/paper/2312.10075/) (2023)
- [MasakhaNEWS: News Topic Classification for African languages](https://mlsanity.com/paper/2304.09972/) (2023)
- [Building African Voices](https://mlsanity.com/paper/2207.00688/) (2022)
- [Beyond Metrics: Evaluating LLMs' Effectiveness in Culturally Nuanced, Low-Resource Real-World Scenarios](https://mlsanity.com/paper/2406.00343/) (2024)
- [Censorship of Online Encyclopedias: Implications for NLP Models](https://mlsanity.com/paper/2101.09294/) (2021)
- [Born With a Silver Spoon? Investigating Socioeconomic Bias in Large Language Models](https://mlsanity.com/paper/2403.14633/) (2024)
- [National Origin Discrimination in Deep-learning-powered Automated Resume Screening](https://mlsanity.com/paper/2307.08624/) (2023)

## Built on by (most cited first, 7 of 7)

- [Break the Checkbox: Challenging Closed-Style Evaluations of Cultural Alignment in LLMs](https://mlsanity.com/paper/2502.08045/) (2025)
- [On the Credibility of Evaluating LLMs using Survey Questions](https://mlsanity.com/paper/2602.04033/) (2026)
- [Improving Cross-Cultural Survey Simulation with Calibrated Value Personas](https://mlsanity.com/paper/2605.16193/) (2026)
- [Toward Cultural Alignment: Human-Centered Evaluation of Multimodal AI Stories Across Five African Communities](https://mlsanity.com/paper/2608.29209/) (2026)
- [LLMs Mirror Country-Specific Gender Patterns If Asked, but Skew Male When Generating Media in Local Languages](https://mlsanity.com/paper/2609.06545/) (2026)
- [Deflecting the Value Compass: Interacting with Large Language Models Temporarily Shifts Human Value Priorities Toward Personal Focus](https://mlsanity.com/paper/2609.25586/) (2026)
- [Measuring Cultural Alignment Beyond the Average: A Framework for Evaluating Maternal-Health LLM Interactions in Indian Contexts](https://mlsanity.com/paper/2610.11586/) (2026)

## Abstract (arXiv)

The training data for LLMs embeds societal values, increasing their familiarity with the language's culture. Our analysis found that 44% of the variance in the ability of GPT-4o to reflect the societal values of a country, as measured by the World Values Survey, correlates with the availability of digital resources in that language. Notably, the error rate was more than five times higher for the languages of the lowest resource compared to the languages of the highest resource. For GPT-4-turbo, this correlation rose to 72%, suggesting efforts to improve the familiarity with the non-English language beyond the web-scraped data. Our study developed one of the largest and most robust datasets in this topic area with 21 country-language pairs, each of which contain 94 survey questions verified by native speakers. Our results highlight the link between LLM performance and digital data availability in target languages. Weaker performance in low-resource languages, especially prominent in the Global South, may worsen digital divides. We discuss strategies proposed to address this, including developing multilingual LLMs from the ground up and enhancing fine-tuning on diverse linguistic datasets, as seen in African language initiatives.