# CulturalBench: A Robust, Diverse, and Challenging Cultural Benchmark by Human-AI CulturalTeaming

arXiv: https://arxiv.org/abs/2410.02677 · HTML: https://mlsanity.com/paper/2410.02677/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 37 references that are in the map.
- First posted on arXiv 3 Oct 2024.

## Builds on (most cited first, 27 of 27)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Generative Verifiers: Reward Modeling as Next-Token Prediction](https://mlsanity.com/paper/2408.15240/) (2024)
- [Dynabench: Rethinking Benchmarking in NLP](https://mlsanity.com/paper/2104.14337/) (2021)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [Low-Resource Languages Jailbreak GPT-4](https://mlsanity.com/paper/2310.02446/) (2023)
- [Challenges and Strategies in Cross-Cultural NLP](https://mlsanity.com/paper/2203.10020/) (2022)
- [Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study](https://mlsanity.com/paper/2303.17466/) (2023)
- [Rethinking Benchmark and Contamination for Language Models with Rephrased Samples](https://mlsanity.com/paper/2311.04850/) (2023)
- [Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data](https://mlsanity.com/paper/2404.14367/) (2024)
- [A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity](https://mlsanity.com/paper/2305.13169/) (2023)
- [BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages](https://mlsanity.com/paper/2406.09948/) (2024)
- [Towards Measuring and Modeling "Culture" in LLMs: A Survey](https://mlsanity.com/paper/2403.15412/) (2024)
- [Probing Pre-Trained Language Models for Cross-Cultural Differences in Values](https://mlsanity.com/paper/2203.13722/) (2022)
- [CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark](https://mlsanity.com/paper/2406.05967/) (2024)
- [Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generation](https://mlsanity.com/paper/2204.01171/) (2022)
- [Best Practices and Lessons Learned on Synthetic Data](https://mlsanity.com/paper/2404.07503/) (2024)
- [Benchmarking Vision Language Models for Cultural Understanding](https://mlsanity.com/paper/2407.10920/) (2024)
- [NLPositionality: Characterizing Design Biases of Datasets and Models](https://mlsanity.com/paper/2306.01943/) (2023)
- [Extracting Cultural Commonsense Knowledge at Scale](https://mlsanity.com/paper/2210.07763/) (2022)
- [Massively Multi-Cultural Knowledge Acquisition & LM Benchmarking](https://mlsanity.com/paper/2402.09369/) (2024)
- [Multilingual Language Models are not Multicultural: A Case Study in Emotion](https://mlsanity.com/paper/2307.01370/) (2023)
- [ProtoQA: A Question Answering Dataset for Prototypical Common-Sense Reasoning](https://mlsanity.com/paper/2005.00771/) (2020)
- [EtiCor: Corpus for Analyzing LLMs for Etiquettes](https://mlsanity.com/paper/2310.18974/) (2023)
- [CaLMQA: Exploring culturally specific long-form question answering across 23 languages](https://mlsanity.com/paper/2406.17761/) (2024)

## Built on by (most cited first, 30 of 30)

- [Can Language Models Reason about Individualistic Human Values and Preferences?](https://mlsanity.com/paper/2410.03868/) (2024)
- [Language Matters: How Do Multilingual Input and Reasoning Paths Affect Large Reasoning Models?](https://mlsanity.com/paper/2505.17407/) (2025)
- [Culture in Action: Evaluating Text-to-Image Models through Social Activities](https://mlsanity.com/paper/2511.05681/) (2025)
- [TALES: A Taxonomy and Analysis of Cultural Representations in LLM-generated Stories](https://mlsanity.com/paper/2511.21322/) (2025)
- [MoCo: A One-Stop Shop for Model Collaboration Research](https://mlsanity.com/paper/2601.21257/) (2026)
- [MMA-ASIA: A Multilingual and Multimodal Alignment Framework for Culturally-Grounded Evaluation](https://mlsanity.com/paper/2510.08608/) (2025)
- [Zhyper: Factorized Hypernetworks for Conditioned LLM Fine-Tuning](https://mlsanity.com/paper/2510.19733/) (2025)
- [Positive Alignment: Artificial Intelligence for Human Flourishing](https://mlsanity.com/paper/2605.10310/) (2026)
- [SEADialogues: A Multilingual Culturally Grounded Multi-turn Dialogue Dataset on Southeast Asian Languages](https://mlsanity.com/paper/2508.07069/) (2025)
- [BLEnD-Vis: Benchmarking Multimodal Cultural Understanding in Vision Language Models](https://mlsanity.com/paper/2510.11178/) (2025)
- [Language Models Entangle Language and Culture](https://mlsanity.com/paper/2601.15337/) (2026)
- [Going PLACES: Participatory Localized Red Teaming for Text-to-Image Safety in the Global South](https://mlsanity.com/paper/2605.19190/) (2026)
- [Learning to Route Languages for Multilingual Policy Optimization](https://mlsanity.com/paper/2605.25360/) (2026)
- [PLURAL: A Global Dataset for Value Alignment](https://mlsanity.com/paper/2607.08034/) (2026)
- [Inference-Time Steering for Cross-Lingual Factual Consistency in LLMs](https://mlsanity.com/paper/2607.19243/) (2026)
- [Reasoning Shapes Alignment: Investigating Cultural Alignment in Large Reasoning Models with Cultural Norms](https://mlsanity.com/paper/2511.13359/) (2025)
- [The Role of Dataset Linguistic Structure in the Cultural Awareness of Large Language Models](https://mlsanity.com/paper/2602.01161/) (2026)
- [Thinking with Comics: Enhancing Multimodal Reasoning through Structured Visual Storytelling](https://mlsanity.com/paper/2602.02453/) (2026)
- [SommBench: Assessing Sommelier Expertise of Language Models](https://mlsanity.com/paper/2603.12117/) (2026)
- [On the Cultural Anachronism and Temporal Reasoning in Vision Language Models](https://mlsanity.com/paper/2605.15071/) (2026)
- [Whose Norms? Disentangling Cultural and Personal Alignment in Large Language Models](https://mlsanity.com/paper/2606.07877/) (2026)
- [ADAGE: A Language-Agnostic Pipeline for Analogical Reasoning Evaluation](https://mlsanity.com/paper/2607.23058/) (2026)
- [NormViz: A Benchmark and Framework for Grounding Multimodal Reasoning in Global Cultures](https://mlsanity.com/paper/2609.06831/) (2026)
- [Beyond Cultural Knowledge: Evaluating Arabic Cultural Appropriateness of Large Language Models](https://mlsanity.com/paper/2609.16006/) (2026)
- [FineWeb-CLaR: Culture, Language, and Region Annotations for Benchmark-Aligned Corpus Auditing](https://mlsanity.com/paper/2609.25298/) (2026)
- [Large-scale factor analysis shows machine intelligence is only partially interpretable](https://mlsanity.com/paper/2609.36515/) (2026)
- [Multi-LLM Collaborative Alignment via Stackelberg Games](https://mlsanity.com/paper/2609.39076/) (2026)
- [CuBEs: Culturally-Situated Behavioral Evaluations and the Limitations of Culture-Blind LLM Judges](https://mlsanity.com/paper/2610.02622/) (2026)
- [Adapting English Quality Classifiers for Multilingual LLM Pretraining Data Selection](https://mlsanity.com/paper/2610.11585/) (2026)
- [Compositional RL: Evaluating and Training LLMs for Collaborative Reasoning](https://mlsanity.com/paper/2610.17456/) (2026)

## Abstract (arXiv)

Robust, diverse, and challenging cultural knowledge benchmarks are essential for measuring our progress towards making LMs that are helpful across diverse cultures. We introduce CulturalBench: a set of 1,696 human-written and human-verified questions to assess LMs' cultural knowledge, covering 45 global regions including underrepresented ones like Bangladesh, Zimbabwe, and Peru. Questions are each verified by five independent annotators and span 17 diverse topics ranging from food preferences to greeting etiquette. We construct CulturalBench using methods inspired by Human-AI Red-Teaming. Compared to human performance (92.4% accuracy), the hard version of CulturalBench is challenging even for the best-performing frontier LMs, ranging from 28.7% to 61.5% in accuracy. We find that LMs often struggle with tricky questions that have multiple correct answers (e.g., What utensils do the Chinese usually use?), revealing a tendency to overfit to a single answer. Our results indicate that GPT-4o substantially outperform other models across cultures, besting local providers (e.g., Mistral on European culture and DeepSeek on Chinese culture). Across the board, models under-perform on questions related to North Africa, South America and Middle East.