mlsanityData to 7 Oct 2026

GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Models

arXiv 2205.12247 ↗First posted 24 May 2022cs.CLcited by 30

Not on the map. Why?

Abstract (arXiv)

Recent work has shown that Pre-trained Language Models (PLMs) store the relational knowledge learned from data and utilize it for performing downstream tasks. However, commonsense knowledge across different regions may vary. For instance, the color of bridal dress is white in American weddings whereas it is red in Chinese weddings. In this paper, we introduce a benchmark dataset, Geo-Diverse Commonsense Multilingual Language Models Analysis (GeoMLAMA), for probing the diversity of the relational knowledge in multilingual PLMs. GeoMLAMA contains 3,125 prompts in English, Chinese, Hindi, Persian, and Swahili, with a wide coverage of concepts shared by people from American, Chinese, Indian, Iranian and Kenyan cultures. We benchmark 11 standard multilingual PLMs on GeoMLAMA. Interestingly, we find that 1) larger multilingual PLMs variants do not necessarily store geo-diverse concepts better than its smaller variant; 2) multilingual PLMs are not intrinsically biased towards knowledge from the Western countries (the United States); 3) the native language of a country may not be the best language to probe its knowledge and 4) a language may better probe knowledge about a non-native country than its native country. Code and data are released at https://github.com/WadeYin9712/GeoMLAMA.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 34 of its 50 references that are in the map. First posted on arXiv 24 May 2022.

Builds on

34 of 50 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  5. 2018Deep contextualized word representationscited by 2,285
  6. 2019Language Models as Knowledge Bases?cited by 1,922
  7. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  8. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  9. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  10. 2019How Can We Know What Language Models Know?cited by 953
  11. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  12. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
22 more
  1. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
  2. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
  3. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  4. 2021Visually Grounded Reasoning across Languages and Culturescited by 188
  5. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  6. 2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
  7. 2021Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologiescited by 128
  8. 2020X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Modelscited by 117
  9. 2020Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Modelscited by 110
  10. 2019Evaluating Commonsense in Pre-trained Language Modelscited by 95
  11. 2021Can Language Models be Biomedical Knowledge Bases?cited by 83
  12. 2021A Survey of Race, Racism, and Anti-Racism in NLPcited by 74
  13. 2021Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoningcited by 64
  14. 2020Do Language Embeddings Capture Scales?cited by 63
  15. 2021Broaden the Vision: Geo-Diverse Visual Commonsense Reasoningcited by 59
  16. 2022Welcome to the Modern World of Pronouns: Identity-Inclusive Natural Language Processing beyond Gendercited by 47
  17. 2022Things not Written in Text: Exploring Spatial Commonsense from Visual Signalscited by 41
  18. 2020RICA: Evaluating Robust Inference Capabilities Based on Commonsense Axiomscited by 28
  19. 2020Towards an Atlas of Cultural Commonsense for Machine Reasoningcited by 28
  20. 2020Advanced Semantics for Commonsense Knowledge Extractioncited by 24
  21. 2021Dataset Geography: Mapping Language Data to Language Userscited by 22
  22. 2021Prix-LM: Pretraining for Multilingual Knowledge Base Constructioncited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 10; 2024: 7; 2025: 7; 2026: 2.

30 papers.

Sort:

By area:

  1. 2023Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Modelscited by 117
  2. 2024CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologiescited by 98
  3. 2022Extracting Cultural Commonsense Knowledge at Scalecited by 76
  4. 2023GPT4GEO: How a Language Model Sees the World's Geographycited by 68
  5. 2024CULTURE-GEN: Revealing Global Cultural Perception in Language Models through Natural Language Promptingcited by 55
  6. 2023Knowledge of cultural moral norms in large language modelscited by 51
  7. 2023Inspecting the Geographical Representativeness of Images from Text-to-Image Modelscited by 44
  8. 2023Benchmarks for Automated Commonsense Reasoning: A Surveycited by 40
  9. 2024IndoCulture: Exploring Geographically-Influenced Cultural Commonsense Reasoning Across Eleven Indonesian Provincescited by 38
  10. 2022Speaking Multiple Languages Affects the Moral Bias of Language Modelscited by 34
  11. 2022Geographic and Geopolitical Biases of Language Modelscited by 29
  12. 2024Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Artcited by 26
18 more
  1. 2023GradientCoin: A Peer-to-Peer Decentralized Large Language Modelscited by 23
  2. 2024Cultural Commonsense Knowledge for Intercultural Dialoguescited by 20
  3. 2025Localized Cultural Knowledge is Conserved and Controllable in Large Language Modelscited by 19
  4. 2023Understanding Finetuning for Factual Knowledge Extraction from Language Modelscited by 12
  5. 2023TaskLAMA: Probing the Complex Task Understanding of Language Modelscited by 12
  6. 2023GIVL: Improving Geographical Inclusivity of Vision-Language Models with Pre-Training Methodscited by 11
  7. 2025On the Consistency of Multilingual Context Utilization in Retrieval-Augmented Generationcited by 10
  8. 2024How Well Do LLMs Identify Cultural Unity in Diversity?cited by 8
  9. 2025SaudiCulture: A Benchmark for Evaluating Large Language Models Cultural Competence within Saudi Arabiacited by 8
  10. 2024Temporal Blind Spots in Large Language Modelscited by 7
  11. 2025Characterizing Bias: Benchmarking Large Language Models in Simplified versus Traditional Chinesecited by 6
  12. 2025The World According to LLMs: How Geographic Origin Influences LLMs' Entity Deduction Capabilitiescited by 5
  13. 2022VIPHY: Probing "Visible" Physical Commonsense Knowledgecited by 4
  14. 2026Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMscited by 3
  15. 2025Camellia: Benchmarking Cultural Biases in LLMs for Asian Languagescited by 2
  16. 2026Matching Tasks to Objectives: Fine-Tuning and Prompt-Tuning Strategies for Encoder-Decoder Pre-trained Language Modelscited by 1
  17. 2023Is a Prestigious Job the same as a Prestigious Country? A Case Study on Multilingual Sentence Embeddings and European Countriescited by 0
  18. 2025Tahakom LLM Guidelines and Recipes: From Pre-training Data to an Arabic LLMcited by 0