# CDEval: A Benchmark for Measuring the Cultural Dimensions of Large Language Models

arXiv: https://arxiv.org/abs/2311.16421 · HTML: https://mlsanity.com/paper/2311.16421/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 27 references that are in the map.
- First posted on arXiv 28 Nov 2023.

## Builds on (most cited first, 18 of 18)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [Baichuan 2: Open Large-scale Language Models](https://mlsanity.com/paper/2309.10305/) (2023)
- [Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca](https://mlsanity.com/paper/2304.08177/) (2023)
- [Evaluating the Moral Beliefs Encoded in LLMs](https://mlsanity.com/paper/2307.14324/) (2023)
- [Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study](https://mlsanity.com/paper/2303.17466/) (2023)
- [Probing Pre-Trained Language Models for Cross-Cultural Differences in Values](https://mlsanity.com/paper/2203.13722/) (2022)
- [CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibility](https://mlsanity.com/paper/2307.09705/) (2023)

## Built on by (most cited first, 23 of 23)

- [Survey of Cultural Awareness in Language Models: Text and Beyond](https://mlsanity.com/paper/2411.00860/) (2024)
- [LLM-GLOBE: A Benchmark Evaluating the Cultural Values Embedded in LLM Output](https://mlsanity.com/paper/2411.06032/) (2024)
- [Hire Your Anthropologist! Rethinking Culture Benchmarks Through an Anthropological Lens](https://mlsanity.com/paper/2510.05931/) (2025)
- [Evaluating Cultural Awareness of LLMs for Yoruba, Malayalam, and English](https://mlsanity.com/paper/2410.01811/) (2024)
- [Meta-Cultural Competence: Climbing the Right Hill of Cultural Awareness](https://mlsanity.com/paper/2502.09637/) (2025)
- [SaudiCulture: A Benchmark for Evaluating Large Language Models Cultural Competence within Saudi Arabia](https://mlsanity.com/paper/2503.17485/) (2025)
- [CAReDiO: Cultural Alignment via Representativeness and Distinctiveness Guided Data Optimization](https://mlsanity.com/paper/2504.08820/) (2025)
- [Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectives](https://mlsanity.com/paper/2506.09656/) (2025)
- [AdAEM: An Adaptively and Automated Extensible Measurement of LLMs' Value Difference](https://mlsanity.com/paper/2505.13531/) (2025)
- [Symbolic or Numerical? Understanding Physics Problem Solving in Reasoning LLMs](https://mlsanity.com/paper/2507.01334/) (2025)
- [AlignCultura: Towards Culturally Aligned Large Language Models?](https://mlsanity.com/paper/2604.19016/) (2026)
- [CommunityBench: Benchmarking Community-Level Alignment across Diverse Groups and Tasks](https://mlsanity.com/paper/2601.13669/) (2026)
- [LiveCultureBench: a Multi-Agent, Multi-Cultural Benchmark for Large Language Models in Dynamic Social Simulations](https://mlsanity.com/paper/2603.01952/) (2026)
- [Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebook](https://mlsanity.com/paper/2604.06210/) (2026)
- [Musical ethnocentrism in Large Language Models](https://mlsanity.com/paper/2501.13720/) (2025)
- [Analyzing Cognitive Differences Among Large Language Models through the Lens of Social Worldview](https://mlsanity.com/paper/2505.01967/) (2025)
- [Conceptual Cultural Index: A Metric for Cultural Specificity via Relative Generality](https://mlsanity.com/paper/2602.09444/) (2026)
- [Toward Culturally Grounded Natural Language Processing](https://mlsanity.com/paper/2603.26013/) (2026)
- [KG-FairDiff: Knowledge Graph-Guided Prompt Refinement for Demographically Fair Text-to-Image Generation](https://mlsanity.com/paper/2606.01282/) (2026)
- [Culturally-Adapted Red-Teaming Across East and Southeast Asian Contexts: A Methodological and Comparative Analysis](https://mlsanity.com/paper/2606.09178/) (2026)
- [GYROval: A Robust Benchmark for Cultural Value Orientation in Large Language Models](https://mlsanity.com/paper/2609.18384/) (2026)
- [FineWeb-CLaR: Culture, Language, and Region Annotations for Benchmark-Aligned Corpus Auditing](https://mlsanity.com/paper/2609.25298/) (2026)
- [Deflecting the Value Compass: Interacting with Large Language Models Temporarily Shifts Human Value Priorities Toward Personal Focus](https://mlsanity.com/paper/2609.25586/) (2026)

## Abstract (arXiv)

As the scaling of Large Language Models (LLMs) has dramatically enhanced their capabilities, there has been a growing focus on the alignment problem to ensure their responsible and ethical use. While existing alignment efforts predominantly concentrate on universal values such as the HHH principle, the aspect of culture, which is inherently pluralistic and diverse, has not received adequate attention. This work introduces a new benchmark, CDEval, aimed at evaluating the cultural dimensions of LLMs. CDEval is constructed by incorporating both GPT-4's automated generation and human verification, covering six cultural dimensions across seven domains. Our comprehensive experiments provide intriguing insights into the culture of mainstream LLMs, highlighting both consistencies and variations across different dimensions and domains. The findings underscore the importance of integrating cultural considerations in LLM development, particularly for applications in diverse cultural settings. Through CDEval, we aim to broaden the horizon of LLM alignment research by including cultural dimensions, thus providing a more holistic framework for the future development and evaluation of LLMs. This benchmark serves as a valuable resource for cultural studies in LLMs, paving the way for more culturally aware and sensitive models.