mlsanityData to 7 Oct 2026

Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayings

arXiv 2309.08591 ↗First posted 15 Sep 2023cs.CLcited by 32

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models (LLMs) are highly adept at question answering and reasoning tasks, but when reasoning in a situational context, human expectations vary depending on the relevant cultural common ground. As languages are associated with diverse cultures, LLMs should also be culturally-diverse reasoners. In this paper, we study the ability of a wide range of state-of-the-art multilingual LLMs (mLLMs) to reason with proverbs and sayings in a conversational context. Our experiments reveal that: (1) mLLMs "know" limited proverbs and memorizing proverbs does not mean understanding them within a conversational context; (2) mLLMs struggle to reason with figurative proverbs and sayings, and when asked to select the wrong answer (instead of asking it to select the correct answer); and (3) there is a "culture gap" in mLLMs when reasoning about proverbs and sayings translated from other languages. We construct and release our evaluation dataset MAPS (MulticultrAl Proverbs and Sayings) for proverb understanding with conversational context for six different languages.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 37 of its 50 references that are in the map. First posted on arXiv 15 Sep 2023.

Builds on

37 of 50 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023ScoNe: Benchmarking Negation Reasoning in Language Models With Fine-Tuning and In-Context Learning30 May

Other references · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2020Extracting Training Data from Large Language Modelscited by 2,144
  7. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  9. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  10. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  11. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  12. 2020Language-agnostic BERT Sentence Embeddingcited by 621
24 more
  1. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  2. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  3. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  4. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 240
  5. 2022Data Contamination: From Memorization to Exploitationcited by 234
  6. 2022Challenges and Strategies in Cross-Cultural NLPcited by 216
  7. 2023Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Studycited by 203
  8. 2021Visually Grounded Reasoning across Languages and Culturescited by 188
  9. 2022Probing Pre-Trained Language Models for Cross-Cultural Differences in Valuescited by 147
  10. 2023We're Afraid Language Models Aren't Modeling Ambiguitycited by 124
  11. 2023SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoningcited by 94
  12. 2023Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptationcited by 79
  13. 2022FLUTE: Figurative Language Understanding through Textual Explanationscited by 78
  14. 2023Language models are not naysayers: An analysis of language models on negation benchmarkscited by 73
  15. 2022A fine-grained comparison of pragmatic language understanding in humans and language modelscited by 69
  16. 2022Testing the Ability of Language Models to Interpret Figurative Languagecited by 67
  17. 2022Understanding Transformer Memorization Recall Through Idiomscited by 62
  18. 2022The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMscited by 57
  19. 2023Benchmarking Machine Translation with Cultural Awarenesscited by 53
  20. 2023Multi-lingual and Multi-cultural Figurative Language Understandingcited by 47
  21. 2022Speaking Multiple Languages Affects the Moral Bias of Language Modelscited by 34
  22. 2022Cross-Lingual Dialogue Dataset Creation via Outline-Based Generationcited by 23
  23. 2023Modeling Cross-Cultural Pragmatic Inference with Codenames Duetcited by 20
  24. 2021ePiC: Employing Proverbs in Context as a Benchmark for Abstract Language Understandingcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 11; 2025: 14; 2026: 5.

32 papers.

Sort:

By area:

  1. 2025Do Multilingual LLMs Think In English?cited by 75
  2. 2024CultureLLM: Incorporating Cultural Differences into Large Language Modelscited by 49
  3. 2025CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countriescited by 40
  4. 2024IndoCulture: Exploring Geographically-Influenced Cultural Commonsense Reasoning Across Eleven Indonesian Provincescited by 38
  5. 2023COPAL-ID: Indonesian Language Reasoning with Local Culture and Nuancescited by 35
  6. 2024Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Artcited by 26
  7. 2024CulturePark: Boosting Cross-cultural Understanding in Large Language Modelscited by 24
  8. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  9. 2025Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMscited by 17
  10. 2024A Survey on Human-Centric LLMscited by 13
  11. 2024On Classification with Large Language Models in Cultural Analyticscited by 9
  12. 2024How Well Do LLMs Identify Cultural Unity in Diversity?cited by 8
20 more
  1. 2024Evaluating Cultural Awareness of LLMs for Yoruba, Malayalam, and Englishcited by 8
  2. 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Modelscited by 7
  3. 2024Navigating the Landscape of Hint Generation Research: From the Past to the Futurecited by 6
  4. 2025Language Specific Knowledge: Do Models Know Better in X than in English?cited by 4
  5. 2025Do Large Language Models Understand Morality Across Cultures?cited by 4
  6. 2025Identity-Aware Large Language Models require Cultural Reasoningcited by 4
  7. 2025Through the Prism of Culture: Evaluating LLMs' Understanding of Indian Subcultures and Traditionscited by 3
  8. 2025Evaluating Large Language Model with Knowledge Oriented Language Specific Simple Question Answeringcited by 3
  9. 2025Exploring Cultural Variations in Moral Judgments with Large Language Modelscited by 3
  10. 2025MCEval: A Dynamic Framework for Fair Multilingual Cultural Evaluation of LLMscited by 3
  11. 2024RoMath: A Mathematical Reasoning Benchmark in Romaniancited by 2
  12. 2025When LLMs Team Up: The Emergence of Collaborative Affective Computingcited by 2
  13. 2026A Unified Framework to Quantify Cultural Intelligence of AIcited by 2
  14. 2026Anthropogenic Regional Adaptation in Multimodal Vision-Language Modelcited by 2
  15. 2023Advancing Perception in Artificial Intelligence through Principles of Cognitive Sciencecited by 1
  16. 2025The Riddle of Reflection: Evaluating Reasoning and Self-Awareness in Multilingual LLMs using Indian Riddlescited by 1
  17. 2025Delving Into the Psychology of Machines: Exploring the Structure of Self-Regulated Learning via LLM-Generated Survey Responsescited by 0
  18. 2026On the Cultural Anachronism and Temporal Reasoning in Vision Language Modelscited by 0
  19. 2026NRITYAM: Language Models Meet Art and Heritage of Dancecited by 0
  20. 2026Evaluating Communicative Success in Machine-Translated Conversationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Easy to Complete, Hard to Choose: Investigating LLM Performance on the ProverbIT Benchmark
  2. 2026MasalBench: A Benchmark for Contextual and Cross-Cultural Understanding of Persian Proverbs in LLMs
  3. 2025CRaFT: An Explanation-Based Framework for Evaluating Cultural Reasoning in Multilingual Language Models
  4. 2025MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMs
  5. 2026mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages?
  6. 2025Disparities in LLM Reasoning Accuracy and Explanations: A Case Study on African American English
  7. 2024A Statistical Analysis of LLMs' Self-Evaluation Using Proverbs
  8. 2025A Survey of Multilingual Reasoning in Language Models