mlsanityData to 7 Oct 2026

BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages

arXiv 2406.09948 ↗First posted 14 Jun 2024cs.CLcited by 171

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) often lack culture-specific knowledge of daily life, especially across diverse regions and non-English languages. Existing benchmarks for evaluating LLMs' cultural sensitivities are limited to a single language or collected from online sources such as Wikipedia, which do not reflect the mundane everyday lifestyles of diverse regions. That is, information about the food people eat for their birthday celebrations, spices they typically use, musical instruments youngsters play, or the sports they practice in school is common cultural knowledge but uncommon in easily collected online sources, especially for underrepresented cultures. To address this issue, we introduce BLEnD, a hand-crafted benchmark designed to evaluate LLMs' everyday knowledge across diverse cultures and languages. BLEnD comprises 52.6k question-answer pairs from 16 countries/regions, in 13 different languages, including low-resource ones such as Amharic, Assamese, Azerbaijani, Hausa, and Sundanese. We construct the benchmark to include two formats of questions: short-answer and multiple-choice. We show that LLMs perform better for cultures that are highly represented online, with a maximum 57.34% difference in GPT-4, the best-performing model, in the short-answer format. For cultures represented by mid-to-high-resource languages, LLMs perform better in their local languages, but for cultures represented by low-resource languages, LLMs perform better in English than the local languages. We make our dataset publicly available at: https://github.com/nlee0212/BLEnD.

Key facts. Cited by 171 papers in the map (data through 7 Oct 2026). Builds on 24 of its 36 references that are in the map. First posted on arXiv 14 Jun 2024.

Builds on

24 of 36 references.

References · 24

  1. 2023Qwen Technical Reportcited by 3,273
  2. 2019Language Models as Knowledge Bases?cited by 1,922
  3. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  4. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  5. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  6. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  7. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  8. 2022Challenges and Strategies in Cross-Cultural NLPcited by 216
  9. 2023Having Beer after Prayer? Measuring Cultural Bias in Large Language Modelscited by 153
  10. 2024Aya 23: Open Weight Releases to Further Multilingual Progresscited by 121
  11. 2022Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generationcited by 106
  12. 2023SeaLLMs -- Large Language Models for Southeast Asiacited by 104
12 more
  1. 2024CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologiescited by 98
  2. 2022Extracting Cultural Commonsense Knowledge at Scalecited by 76
  3. 2024CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Koreancited by 65
  4. 2024Massively Multi-Cultural Knowledge Acquisition & LM Benchmarkingcited by 63
  5. 2024Evaluating Gender Bias in Large Language Models via Chain-of-Thought Promptingcited by 49
  6. 2023HAE-RAE Bench: Evaluation of Korean Knowledge in Language Modelscited by 47
  7. 2023Exploring Cross-Cultural Differences in English Hate Speech Annotations: From Dataset Construction to Analysiscited by 44
  8. 2024IndoCulture: Exploring Geographically-Influenced Cultural Commonsense Reasoning Across Eleven Indonesian Provincescited by 38
  9. 2024Can LLM Generate Culturally Relevant Commonsense QA Data? Case Study in Indonesian and Sundanesecited by 36
  10. 2022StereoKG: Data-Driven Knowledge Graph Construction for Cultural Knowledge and Stereotypescited by 14
  11. 2014Big Questions for Social Media Big Data: Representativeness, Validity and Other Methodological Pitfallscited by 11
  12. 2023A Predictive Factor Analysis of Social Biases and Task-Performance in Pretrained Masked Language Modelscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 74; 2026: 86.

171 papers.

Sort:

By area:

  1. 2024Qwen2.5 Technical Reportcited by 4,713
  2. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  3. 2024Survey of Cultural Awareness in Language Models: Text and Beyondcited by 101
  4. 2024INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledgecited by 78
  5. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 51
  6. 2024AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMscited by 46
  7. 2025CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countriescited by 40
  8. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 34
  9. 2025Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Datasetcited by 32
  10. 2024CaLMQA: Exploring culturally specific long-form question answering across 23 languagescited by 31
  11. 2024NativQA: Multilingual Culturally-Aligned Natural Query for LLMscited by 30
  12. 2025An Evaluation of Cultural Value Alignment in LLMcited by 28
38 more
  1. 2024CulturalBench: A Robust, Diverse, and Challenging Cultural Benchmark by Human-AI CulturalTeamingcited by 27
  2. 2025Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMscited by 17
  3. 2025The Case for "Thick Evaluations" of Cultural Representation in AIcited by 16
  4. 2024ProverbEval: Exploring LLM Evaluation Challenges for Low-resource Language Understandingcited by 15
  5. 2025Palm: A Culturally Inclusive and Linguistically Diverse Dataset for Arabic LLMscited by 15
  6. 2025CARE: Multilingual Human Preference Learning for Cultural Awarenesscited by 15
  7. 2025Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Aheadcited by 15
  8. 2025Biased Tales: Cultural and Topic Bias in Generating Children's Storiescited by 15
  9. 2025Diffusion Models Through a Global Lens: Are They Culturally Inclusive?cited by 14
  10. 2025CulFiT: A Fine-grained Cultural-aware LLM Training Paradigm via Multilingual Critique Data Synthesiscited by 14
  11. 2026Tiny Aya: Bridging Scale and Multilingual Depthcited by 14
  12. 2025MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMscited by 13
  13. 2025DRISHTIKON: A Multimodal Multilingual Benchmark for Testing Language Models' Understanding on Indian Culturecited by 13
  14. 2025Hire Your Anthropologist! Rethinking Culture Benchmarks Through an Anthropological Lenscited by 13
  15. 2025CultureScope: A Dimensional Lens for Probing Cultural Understanding in LLMscited by 11
  16. 2025Culture is Everywhere: A Call for Intentionally Cultural Evaluationcited by 10
  17. 2025TALES: A Taxonomy and Analysis of Cultural Representations in LLM-generated Storiescited by 10
  18. 2025From Surveys to Narratives: Rethinking Cultural Value Adaptation in LLMscited by 9
  19. 2024Evaluating Cultural Awareness of LLMs for Yoruba, Malayalam, and Englishcited by 8
  20. 2025TiEBe: Tracking Language Model Recall of Notable Worldwide Events Through Timecited by 8
  21. 2025PerCul: A Story-Driven Cultural Evaluation of LLMs in Persiancited by 8
  22. 2025Meta-Cultural Competence: Climbing the Right Hill of Cultural Awarenesscited by 8
  23. 2025Crowdsource, Crawl, or Generate? Creating SEA-VL, a Multicultural Vision-Language Dataset for Southeast Asiacited by 8
  24. 2025SaudiCulture: A Benchmark for Evaluating Large Language Models Cultural Competence within Saudi Arabiacited by 8
  25. 2025Do You Know About My Nation? Investigating Multilingual Language Models' Cultural Literacy Through Factual Knowledgecited by 8
  26. 2025Evaluating Polish linguistic and cultural competency in large language modelscited by 6
  27. 2025LLM Alignment for the Arabs: A Homogenous Culture or Diverse Ones?cited by 6
  28. 2025FailureSensorIQ: A Multi-Choice QA Dataset for Understanding Sensor Relationships and Failure Modescited by 6
  29. 2025Nunchi-Bench: Benchmarking Language Models on Cultural Reasoning with a Focus on Korean Superstitioncited by 6
  30. 2025CultureSynth: A Hierarchical Taxonomy-Guided and Retrieval-Augmented Framework for Cultural Question-Answer Synthesiscited by 6
  31. 2025CML-Bench: A Framework for Evaluating and Enhancing LLM-Powered Movie Scripts Generationcited by 6
  32. 2025Towards Geo-Culturally Grounded LLM Generationscited by 5
  33. 2025CAReDiO: Cultural Alignment via Representativeness and Distinctiveness Guided Data Optimizationcited by 5
  34. 2025WorldView-Bench: A Benchmark for Evaluating Global Cultural Perspectives in Large Language Modelscited by 5
  35. 2025AI Agent Behavioral Sciencecited by 5
  36. 2025Building Benchmarks from the Ground Up: Community-Centered Evaluation of LLMs in Healthcare Chatbot Settingscited by 5
  37. 2025PoETa v2: Toward More Robust Evaluation of Large Language Models in Portuguesecited by 5
  38. 2026MoCo: A One-Stop Shop for Model Collaboration Researchcited by 5

50 most cited of 171 All 171 (JSON)