mlsanityData to 14 Oct 2026

Probing Pre-Trained Language Models for Cross-Cultural Differences in Values

arXiv 2203.13722 ↗First posted 25 Mar 2022cs.CLcited by 149

Not on the map. Why?

Abstract (arXiv)

Language embeds information about social, cultural, and political values people hold. Prior work has explored social and potentially harmful biases encoded in Pre-Trained Language models (PTLMs). However, there has been no systematic study investigating how values embedded in these models vary across cultures. In this paper, we introduce probes to study which values across cultures are embedded in these models, and whether they align with existing theories and cross-cultural value surveys. We find that PTLMs capture differences in values across cultures, but those only weakly align with established value surveys. We discuss implications of using mis-aligned models in cross-cultural settings, as well as ways of aligning PTLMs with value surveys.

Key facts. Cited by 149 papers in the map (data through 14 Oct 2026). Builds on 40 of its 76 references that are in the map. First posted on arXiv 25 Mar 2022.

Builds on

40 of 76 references.

References · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  5. 2019Language Models as Knowledge Bases?cited by 1,936
  6. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  7. 2015Teaching Machines to Read and Comprehendcited by 1,049
  8. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  9. 2016Semantics derived automatically from language corpora contain human-like biasescited by 901
  10. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  11. 2020Aligning AI With Shared Human Valuescited by 723
  12. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
28 more
  1. 2019On Measuring Social Biases in Sentence Encoderscited by 398
  2. 2020Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditingcited by 384
  3. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 380
  4. 2020Social Chemistry 101: Learning to Reason about Social and Moral Normscited by 255
  5. 2017Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypescited by 253
  6. 2022Challenges and Strategies in Cross-Cultural NLPcited by 218
  7. 2021Visually Grounded Reasoning across Languages and Culturescited by 189
  8. 2021Word Alignment by Fine-tuning Embeddings on Parallel Corporacited by 186
  9. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 181
  10. 2020Detecting Emergent Intersectional Biases: Contextualized Word Embeddings Contain a Distribution of Human-like Biasescited by 167
  11. 2019Do NLP Models Know Numbers? Probing Numeracy in Embeddingscited by 159
  12. 2020Probing Pretrained Language Models for Lexical Semanticscited by 156
  13. 2020Intrinsic Bias Metrics Do Not Correlate with Application Biascited by 148
  14. 2020Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequencescited by 135
  15. 2020X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Modelscited by 117
  16. 2022The Ghost in the Machine has an American accent: value conflict in GPT-3cited by 105
  17. 2021A Survey on Gender Bias in Natural Language Processingcited by 89
  18. 2021What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpuscited by 80
  19. 2021Unmasking the Mask -- Evaluating Social Biases in Masked Language Modelscited by 60
  20. 2021Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned Language Modelscited by 57
  21. 2020On the Interplay Between Fine-tuning and Sentence-level Probing for Linguistic Knowledge in Pre-trained Transformerscited by 39
  22. 2021Identifying Morality Frames in Political Tweets using Relational Learningcited by 26
  23. 2021A Word on Machine Ethics: A Response to Jiang et al. (2021)cited by 23
  24. 2021Quantifying Gender Bias Towards Politicians in Cross-Lingual Language Modelscited by 15
  25. 2018Socioeconomic Dependencies of Linguistic Patterns in Twitter: A Multivariate Analysiscited by 9
  26. 2021(Mis)alignment Between Stance Expressed in Social Media Data and Public Opinion Surveyscited by 8
  27. 2019Wikipedia Cultural Diversity Dataset: A Complete Cartography for 300 Language Editionscited by 4
  28. 2021Using Sociolinguistic Variables to Reveal Changing Attitudes Towards Sexuality and Gendercited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 27; 2024: 57; 2025: 33; 2026: 29.

149 papers.

Sort:

By area:

  1. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 437
  2. 2023Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Studycited by 206
  3. 2024Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviewscited by 164
  4. 2023Having Beer after Prayer? Measuring Cultural Bias in Large Language Modelscited by 153
  5. 2023Cultural Bias and Cultural Alignment of Large Language Modelscited by 149
  6. 2024Investigating Cultural Alignment of Large Language Modelscited by 143
  7. 2023Cultural Alignment in Large Language Models: An Explanatory Analysis Based on Hofstede's Cultural Dimensionscited by 112
  8. 2024CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologiescited by 98
  9. 2023NLPositionality: Characterizing Design Biases of Datasets and Modelscited by 94
  10. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  11. 2022Moral Mimicry: Large Language Models Produce Moral Rationalizations Tailored to Political Identitycited by 74
  12. 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harmcited by 70
38 more
  1. 2024WorldValuesBench: A Large-Scale Benchmark Dataset for Multi-Cultural Value Awareness of Language Modelscited by 60
  2. 2024AI Suggestions Homogenize Writing Toward Western Styles and Diminish Cultural Nuancescited by 60
  3. 2024DailyDilemmas: Revealing Value Preferences of LLMs with Quandaries of Daily Lifecited by 59
  4. 2024Are Large Language Models Consistent over Value-laden Questions?cited by 58
  5. 2024Beyond prompt brittleness: Evaluating the reliability and consistency of political worldviews in LLMscited by 55
  6. 2023Multilingual Language Models are not Multicultural: A Case Study in Emotioncited by 54
  7. 2025Specializing Large Language Models to Simulate Survey Response Distributions for Global Populationscited by 53
  8. 2023Knowledge of cultural moral norms in large language modelscited by 51
  9. 2023Value FULCRA: Mapping Large Language Models to the Multidimensional Spectrum of Basic Human Valuescited by 50
  10. 2023Large Language Models as Superpositions of Cultural Perspectivescited by 49
  11. 2023From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Modelscited by 45
  12. 2025Culture is Not Trivia: Sociocultural Theory for Cultural NLPcited by 40
  13. 2025Multilingual != Multicultural: Evaluating Gaps Between Multilingual Capabilities and Cultural Alignment in LLMscited by 39
  14. 2024Operationalizing Contextual Integrity in Privacy-Conscious Assistantscited by 37
  15. 2024Evaluating Cultural Adaptability of a Large Language Model via Simulation of Synthetic Personascited by 37
  16. 2024An image speaks a thousand words, but can everyone listen? On image transcreation for cultural relevancecited by 36
  17. 2024MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMscited by 35
  18. 2022Speaking Multiple Languages Affects the Moral Bias of Language Modelscited by 34
  19. 2023This Land is {Your, My} Land: Evaluating Geopolitical Biases in Language Modelscited by 32
  20. 2023Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayingscited by 32
  21. 2024Self-Pluralising Culture Alignment for Large Language Modelscited by 31
  22. 2024CulturalBench: A Robust, Diverse, and Challenging Cultural Benchmark by Human-AI CulturalTeamingcited by 30
  23. 2024A Robot Walks into a Bar: Can Language Models Serve as Creativity Support Tools for Comedy? An Evaluation of LLMs' Humour Alignment with Comedianscited by 29
  24. 2024What Features in Prompts Jailbreak LLMs? Investigating the Mechanisms Behind Attackscited by 29
  25. 2024CulturalTeaming: AI-Assisted Interactive Red-Teaming for Challenging LLMs' (Lack of) Multicultural Knowledgecited by 28
  26. 2025An Evaluation of Cultural Value Alignment in LLMcited by 28
  27. 2024LLM-GLOBE: A Benchmark Evaluating the Cultural Values Embedded in LLM Outputcited by 27
  28. 2023Denevil: Towards Deciphering and Navigating the Ethical Values of Large Language Models via Instruction Learningcited by 26
  29. 2025Break the Checkbox: Challenging Closed-Style Evaluations of Cultural Alignment in LLMscited by 26
  30. 2024SeeGULL Multilingual: a Dataset of Geo-Culturally Situated Stereotypescited by 24
  31. 2024DOSA: A Dataset of Social Artifacts from Different Indian Geographical Subculturescited by 24
  32. 2024Extrinsic Evaluation of Cultural Competence in Large Language Modelscited by 24
  33. 2023CDEval: A Benchmark for Measuring the Cultural Dimensions of Large Language Modelscited by 23
  34. 2023A Material Lens on Coloniality in NLPcited by 21
  35. 2025Multiple LLM Agents Debate for Equitable Cultural Alignmentcited by 21
  36. 2024High-Dimension Human Value Representation in Large Language Modelscited by 20
  37. 2025CASE-Bench: Context-Aware SafEty Benchmark for Large Language Modelscited by 20
  38. 2025Localized Cultural Knowledge is Conserved and Controllable in Large Language Modelscited by 20

50 most cited of 149 All 149 (JSON)