# Can LLMs Grasp Implicit Cultural Values? Benchmarking LLMs' Cultural Intelligence with CQ-Bench

arXiv: https://arxiv.org/abs/2504.01127 · HTML: https://mlsanity.com/paper/2504.01127/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 43 references that are in the map.
- First posted on arXiv 1 Apr 2025.

## Builds on (most cited first, 26 of 26)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages](https://mlsanity.com/paper/2406.09948/) (2024)
- [Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks](https://mlsanity.com/paper/2010.08240/) (2020)
- [The Ghost in the Machine has an American accent: value conflict in GPT-3](https://mlsanity.com/paper/2203.07785/) (2022)
- [SeaLLMs -- Large Language Models for Southeast Asia](https://mlsanity.com/paper/2312.00738/) (2023)
- [CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologies](https://mlsanity.com/paper/2404.15238/) (2024)
- [CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean](https://mlsanity.com/paper/2403.06412/) (2024)
- [CULTURE-GEN: Revealing Global Cultural Perception in Language Models through Natural Language Prompting](https://mlsanity.com/paper/2404.10199/) (2024)
- [How Well Do LLMs Represent Values Across Cultures? Empirical Analysis of LLM Responses Based on Hofstede Cultural Dimensions](https://mlsanity.com/paper/2406.14805/) (2024)
- [NormSAGE: Multi-Lingual Multi-Cultural Norm Discovery from Conversations On-the-Fly](https://mlsanity.com/paper/2210.08604/) (2022)
- [Can LLM Generate Culturally Relevant Commonsense QA Data? Case Study in Indonesian and Sundanese](https://mlsanity.com/paper/2402.17302/) (2024)
- [Taiwan LLM: Bridging the Linguistic Divide with a Culturally Aligned Language Model](https://mlsanity.com/paper/2311.17487/) (2023)
- [CoSER: A Comprehensive Literary Dataset and Framework for Training and Evaluating LLM Role-Playing and Persona Simulation](https://mlsanity.com/paper/2502.09082/) (2025)
- [NormDial: A Comparable Bilingual Synthetic Dialog Dataset for Modeling Social Norm Adherence and Violation](https://mlsanity.com/paper/2310.14563/) (2023)
- [CLAVE: An Adaptive Framework for Evaluating Values of LLM Generated Responses](https://mlsanity.com/paper/2407.10725/) (2024)
- [Cultural Value Differences of LLMs: Prompt, Language, and Model Size](https://mlsanity.com/paper/2407.16891/) (2024)
- [PerCul: A Story-Driven Cultural Evaluation of LLMs in Persian](https://mlsanity.com/paper/2502.07459/) (2025)
- [Meta-Cultural Competence: Climbing the Right Hill of Cultural Awareness](https://mlsanity.com/paper/2502.09637/) (2025)
- [ValueDCG: Measuring Comprehensive Human Value Understanding Ability of Language Models](https://mlsanity.com/paper/2310.00378/) (2023)
- [Enhancing LLM-Based Human-Robot Interaction with Nuances for Diversity Awareness](https://mlsanity.com/paper/2406.17531/) (2024)
- [LLM-Human Pipeline for Cultural Context Grounding of Conversations](https://mlsanity.com/paper/2410.13727/) (2024)

## Built on by (most cited first, 3 of 3)

- [VIRAASAT: Traversing Novel Paths for Indian Cultural Reasoning](https://mlsanity.com/paper/2602.18429/) (2026)
- [Learning the Topic, Not the Language: How LLMs Classify Online Immigration Discourse Across Languages](https://mlsanity.com/paper/2508.06435/) (2025)
- [PALMs: Using Multi Construct-Grounded Rationales for Modeling Population Preferences in LLMs](https://mlsanity.com/paper/2608.01458/) (2026)

## Abstract (arXiv)

Cultural Intelligence (CQ) refers to the ability to understand unfamiliar cultural contexts, a crucial skill for large language models (LLMs) to effectively engage with globally diverse users. Existing studies often focus on explicitly stated cultural norms, but fail to capture the subtle, implicit values that are common in daily conversation. To address this gap, we introduce CQBench, a benchmark specifically designed to assess LLMs' capability to infer implicit cultural values from natural conversational contexts. CQBench consists of multi character conversation based stories using values from the World Value Survey and the GlobalOpinions, with topics including ethical, religious, social, etc. Our automatic dataset construction pipeline integrates rigorous validation procedures (incorporation, consistency, and implicitness checks), achieving a 94.5% human model agreement in the final validation. To leverage CQBench data, we design three tasks of increasing complexity: attitude detection, value selection, and value extraction. These tasks evaluate whether models can detect attitude and recognize values embedded within natural dialogues rather than relying on explicit cultural knowledge. We find that while frontier models like o1 reach human level performance in value selection (0.809 F1), they still fall short in nuanced attitude detection (0.622 F1). Notably, finetuning a smaller LLaMA-3.2-3B on only 500 culturally rich examples improves performance by over 10%, even outperforming o3-mini in some cases. Using CQ-Bench, we provide insights into the current challenges in LLMs' CQ research and suggest practical pathways for enhancing LLMs' cross-cultural reasoning abilities.