mlsanityData to 14 Oct 2026

Having Beer after Prayer? Measuring Cultural Bias in Large Language Models

arXiv 2305.14456 ↗First posted 23 May 2023cs.CL cs.AI cs.LGcited by 153

Not on the map. Why?

Abstract (arXiv)

As the reach of large language models (LMs) expands globally, their ability to cater to diverse cultural contexts becomes crucial. Despite advancements in multilingual capabilities, models are not designed with appropriate cultural nuances. In this paper, we show that multilingual and Arabic monolingual LMs exhibit bias towards entities associated with Western culture. We introduce CAMeL, a novel resource of 628 naturally-occurring prompts and 20,368 entities spanning eight types that contrast Arab and Western cultures. CAMeL provides a foundation for measuring cultural biases in LMs through both extrinsic and intrinsic evaluations. Using CAMeL, we examine the cross-cultural performance in Arabic of 16 different LMs on tasks such as story generation, NER, and sentiment analysis, where we find concerning cases of stereotyping and cultural unfairness. We further test their text-infilling performance, revealing the incapability of appropriate adaptation to Arab cultural contexts. Finally, we analyze 6 Arabic pre-training corpora and find that commonly used sources such as Wikipedia may not be best suited to build culturally aware LMs, if used as they are without adjustment. We will make CAMeL publicly available at: https://github.com/tareknaous/camel

Key facts. Cited by 153 papers in the map (data through 14 Oct 2026). Builds on 61 of its 125 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

61 of 125 references.

References · 61

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  7. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  8. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  9. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  10. 2016Semantics derived automatically from language corpora contain human-like biasescited by 901
  11. 2023Whose Opinions Do Language Models Reflect?cited by 773
  12. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
38 more
  1. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  2. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 659
  3. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 541
  4. 2019On Measuring Social Biases in Sentence Encoderscited by 398
  5. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 377
  6. 2020AraBERT: Transformer-based Model for Arabic Language Understandingcited by 362
  7. 2023MEGA: Multilingual Evaluation of Generative AIcited by 293
  8. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 264
  9. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 260
  10. 2022Challenges and Strategies in Cross-Cultural NLPcited by 218
  11. 2023Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Studycited by 206
  12. 2021Societal Biases in Language Generation: Progress and Challengescited by 202
  13. 2023The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientationcited by 189
  14. 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
  15. 2020Detecting Emergent Intersectional Biases: Contextualized Word Embeddings Contain a Distribution of Human-like Biasescited by 167
  16. 2020ARBERT & MARBERT: Deep Bidirectional Transformers for Arabiccited by 166
  17. 2022"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Datasetcited by 162
  18. 2022Probing Pre-Trained Language Models for Cross-Cultural Differences in Valuescited by 149
  19. 2020Intrinsic Bias Metrics Do Not Correlate with Application Biascited by 148
  20. 2022mGPT: Few-Shot Learners Go Multilingualcited by 142
  21. 2019Assessing Social and Intersectional Biases in Contextualized Word Representationscited by 135
  22. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  23. 2021The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Modelscited by 119
  24. 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
  25. 2022The Ghost in the Machine has an American accent: value conflict in GPT-3cited by 105
  26. 2021Challenges in Automated Debiasing for Toxic Language Detectioncited by 96
  27. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  28. 2021A Survey on Gender Bias in Natural Language Processingcited by 89
  29. 2021Mitigating Language-Dependent Ethnic Bias in BERTcited by 87
  30. 2021Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metricscited by 83
  31. 2022Extracting Cultural Commonsense Knowledge at Scalecited by 76
  32. 2021Towards generalisable hate speech detection: a review on obstacles and solutionscited by 66
  33. 2020Measuring Social Biases in Grounded Vision and Language Embeddingscited by 61
  34. 2021Unmasking the Mask -- Evaluating Social Biases in Masked Language Modelscited by 60
  35. 2022Gender Bias in Masked Language Models for Multiple Languagescited by 60
  36. 2020AraGPT2: Pre-Trained Transformer for Arabic Language Generationcited by 55
  37. 2022Re-contextualizing Fairness in NLP: The Case of Indiacited by 55
  38. 2022ORCA: A Challenging Benchmark for Arabic Language Understandingcited by 47

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 46; 2025: 47; 2026: 54.

153 papers.

Sort:

By area:

  1. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 437
  2. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 212
  3. 2024BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languagescited by 173
  4. 2024Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviewscited by 164
  5. 2023Cultural Bias and Cultural Alignment of Large Language Modelscited by 149
  6. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaborationcited by 113
  7. 2024Survey of Cultural Awareness in Language Models: Text and Beyondcited by 103
  8. 2024CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologiescited by 98
  9. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 84
  10. 2024Towards Conversational Diagnostic AIcited by 79
  11. 2025Do Multilingual LLMs Think In English?cited by 78
  12. 2024Unintended Impacts of LLM Alignment on Global Representationcited by 66
38 more
  1. 2024CULTURE-GEN: Revealing Global Cultural Perception in Language Models through Natural Language Promptingcited by 55
  2. 2024ALLaM: Large Language Models for Arabic and Englishcited by 55
  3. 2024CultureLLM: Incorporating Cultural Differences into Large Language Modelscited by 50
  4. 2024AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMscited by 46
  5. 2025CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countriescited by 42
  6. 2024Evaluating Cultural Adaptability of a Large Language Model via Simulation of Synthetic Personascited by 37
  7. 2024Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirementscited by 36
  8. 2024CaLMQA: Exploring culturally specific long-form question answering across 23 languagescited by 31
  9. 2024Beyond Aesthetics: Cultural Competence in Text-to-Image Modelscited by 30
  10. 2024NativQA: Multilingual Culturally-Aligned Natural Query for LLMscited by 30
  11. 2024CulturalTeaming: AI-Assisted Interactive Red-Teaming for Challenging LLMs' (Lack of) Multicultural Knowledgecited by 28
  12. 2025An Evaluation of Cultural Value Alignment in LLMcited by 28
  13. 2024Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Artcited by 26
  14. 2024MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignmentcited by 25
  15. 2024CulturePark: Boosting Cross-cultural Understanding in Large Language Modelscited by 24
  16. 2024BertaQA: How Much Do Language Models Know About Local Culture?cited by 23
  17. 2024Assessing Dialect Fairness and Robustness of Large Language Models in Reasoning Taskscited by 22
  18. 2024High-Dimension Human Value Representation in Large Language Modelscited by 20
  19. 2025Localized Cultural Knowledge is Conserved and Controllable in Large Language Modelscited by 20
  20. 2023BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Modelscited by 18
  21. 2024Multi-FAct: Assessing Factuality of Multilingual LLMs using FActScorecited by 18
  22. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  23. 2024See It from My Perspective: How Language Affects Cultural Bias in Image Understandingcited by 17
  24. 2024Whose Morality Do They Speak? Unraveling Cultural Bias in Multilingual Language Modelscited by 17
  25. 2023Navigating Cultural Chasms: Exploring and Unlocking the Cultural POV of Text-To-Image Modelscited by 16
  26. 2024Cross-Care: Assessing the Healthcare Implications of Pre-training Data on Language Model Biascited by 15
  27. 2025MIRROR: Multimodal Cognitive Reframing Therapy for Rolling with Resistancecited by 13
  28. 2025MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMscited by 13
  29. 2024Representation Bias in Political Sample Simulations with Large Language Modelscited by 10
  30. 2024CamelEval: Advancing Culturally Aligned Arabic Language Models and Benchmarkscited by 10
  31. 2025A Comprehensive Analysis of Large Language Model Outputs: Similarity, Diversity, and Biascited by 10
  32. 2025The Multilingual Divide and Its Impact on Global AI Safetycited by 10
  33. 2025Lost in Cultural Translation: Do LLMs Struggle with Math Across Cultural Contexts?cited by 9
  34. 2025SPARTA ALIGNMENT: Collectively Aligning Multiple Language Models through Combatcited by 9
  35. 2024Evaluating Cultural Awareness of LLMs for Yoruba, Malayalam, and Englishcited by 8
  36. 2024Attributing Culture-Conditioned Generations to Pretraining Corporacited by 8
  37. 2025Pragmatic Inference Chain (PIC) Improving LLMs' Reasoning of Authentic Implicit Toxic Languagecited by 8
  38. 2025SaudiCulture: A Benchmark for Evaluating Large Language Models Cultural Competence within Saudi Arabiacited by 8

50 most cited of 153 All 153 (JSON)