mlsanityData to 7 Oct 2026

CRaFT: An Explanation-Based Framework for Evaluating Cultural Reasoning in Multilingual Language Models

arXiv 2510.14014 ↗First posted 15 Oct 2025cs.CLcited by 1

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Correct answers do not necessarily reflect cultural understanding. We introduce CRaFT, an explanation-based multilingual evaluation framework designed to assess how large language models (LLMs) reason across cultural contexts. Rather than scoring outputs solely based on accuracy, CRaFT evaluates model explanations using four interpretable metrics: Cultural Fluency, Deviation, Consistency, and Linguistic Adaptation. We apply the framework to 50 culturally grounded questions from the World Values Survey, translated into Arabic, Bengali, and Spanish, and evaluate three models (GPT, DeepSeek, and FANAR) across over 2,100 answer-explanation pairs. Results reveal significant cross-lingual variation in reasoning: Arabic reduces fluency, Bengali enhances it, and Spanish remains largely stable. While GPT adapts more effectively across languages, it exhibits lower consistency; FANAR shows stable but rigid reasoning. These findings suggest that cultural awareness in LLMs is not intrinsic but emerges through linguistic framing. CRaFT offers a new lens for evaluating cross-cultural reasoning in multilingual settings, providing actionable insights for building culturally adaptive language models.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 28 of its 40 references that are in the map. First posted on arXiv 15 Oct 2025.

Builds on

28 of 40 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Culturally-Grounded Chain-of-Thought (CG-CoT):Enhancing LLM Performance on Culturally-Specific Tasks in Low-Resource Languages1 Jun
  2. 2025IRLBench: A Multi-modal, Culturally Grounded, Parallel Irish-English Benchmark for Open-Ended LLM Reasoning Evaluation16 May
  3. 2024ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic20 Feb

Other references · 25

  1. 2023Whose Opinions Do Language Models Reflect?cited by 759
  2. 2015Yara Parser: A Fast and Accurate Dependency Parsercited by 110
  3. 2024Are self-explanations from Large Language Models faithful?cited by 101
  4. 2024WorldValuesBench: A Large-Scale Benchmark Dataset for Multi-Cultural Value Awareness of Language Modelscited by 59
  5. 2025Randomness, Not Representation: The Unreliability of Evaluating Cultural Alignment in LLMscited by 54
  6. 2024NormAd: A Framework for Measuring the Cultural Adaptability of Large Language Modelscited by 47
  7. 2022IndicNLG Benchmark: Multilingual Datasets for Diverse NLG Tasks in Indic Languagescited by 42
  8. 2022BanglaNLG and BanglaT5: Benchmarks and Resources for Evaluating Low-Resource Natural Language Generation in Banglacited by 40
  9. 2025An Evaluation of Cultural Value Alignment in LLMcited by 28
  10. 2025MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languagescited by 22
  11. 2024Spanish and LLM Benchmarks: is MMLU Lost in Translation?cited by 20
  12. 2024Arabic Dataset for LLM Safeguard Evaluationcited by 17
13 more
  1. 2025Cultural Learning-Based Culture Adaptation of Language Modelscited by 16
  2. 2025CulFiT: A Fine-grained Cultural-aware LLM Training Paradigm via Multilingual Critique Data Synthesiscited by 14
  3. 2025MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMscited by 13
  4. 2025A Causal Lens for Evaluating Faithfulness Metricscited by 12
  5. 2025CultureScope: A Dimensional Lens for Probing Cultural Understanding in LLMscited by 11
  6. 2024Swan and ArabicMTEB: Dialect-Aware, Arabic-Centric, Cross-Lingual, and Cross-Cultural Embedding Models and Benchmarkscited by 10
  7. 2025TituLLMs: A Family of Bangla LLMs with Comprehensive Benchmarkingcited by 9
  8. 2025SaudiCulture: A Benchmark for Evaluating Large Language Models Cultural Competence within Saudi Arabiacited by 8
  9. 2024Comparing Explanation Faithfulness between Multilingual and Monolingual Fine-tuned Language Modelscited by 6
  10. 2025BnMMLU: Measuring Massive Multitask Language Understanding in Bengalicited by 6
  11. 2024Socio-Culturally Aware Evaluation Framework for LLM-Based Content Moderationcited by 5
  12. 2025MCEval: A Dynamic Framework for Fair Multilingual Cultural Evaluation of LLMscited by 3
  13. 2025From Words to Proverbs: Evaluating LLMs Linguistic and Cultural Competence in Saudi Dialects with Abshercited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Unmasking the Factual-Conceptual Gap in Persian Language Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026XCR-Bench: Benchmarking Cross-Cultural Reasoning in LLMs via Culture-Specific Items and Hall's Triad
  2. 2025CulTrace: Tracing Internal Cultural Reasoning in Large Language Models
  3. 2025CURE: Cultural Understanding and Reasoning Evaluation - A Framework for "Thick" Culture Alignment Evaluation in LLMs
  4. 2023Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayings
  5. 2026x1: Learning to Think Adaptively Across Languages and Cultures
  6. 2025Beg to Differ: Understanding Reasoning-Answer Misalignment Across Languages
  7. 2025MMA-ASIA: A Multilingual and Multimodal Alignment Framework for Culturally-Grounded Evaluation
  8. 2025Do Large Language Models Truly Understand Cross-cultural Differences?