# GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Models

arXiv: https://arxiv.org/abs/2205.12247 · HTML: https://mlsanity.com/paper/2205.12247/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 50 references that are in the map.
- First posted on arXiv 24 May 2022.

## Builds on (most cited first, 30 of 34)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [COMET: Commonsense Transformers for Automatic Knowledge Graph Construction](https://mlsanity.com/paper/1906.05317/) (2019)
- [XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning](https://mlsanity.com/paper/2005.00333/) (2020)
- [Factual Probing Is [MASK]: Learning vs. Learning to Recall](https://mlsanity.com/paper/2104.05240/) (2021)
- [Visually Grounded Reasoning across Languages and Cultures](https://mlsanity.com/paper/2109.13238/) (2021)
- [oLMpics -- On what Language Model Pre-training Captures](https://mlsanity.com/paper/1912.13283/) (2019)
- [Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models](https://mlsanity.com/paper/2102.00894/) (2021)
- [Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies](https://mlsanity.com/paper/2108.12084/) (2021)
- [X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models](https://mlsanity.com/paper/2010.06189/) (2020)
- [Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Models](https://mlsanity.com/paper/2005.00683/) (2020)
- [Evaluating Commonsense in Pre-trained Language Models](https://mlsanity.com/paper/1911.11931/) (2019)
- [Can Language Models be Biomedical Knowledge Bases?](https://mlsanity.com/paper/2109.07154/) (2021)
- [A Survey of Race, Racism, and Anti-Racism in NLP](https://mlsanity.com/paper/2106.11410/) (2021)
- [Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoning](https://mlsanity.com/paper/2106.06937/) (2021)
- [Do Language Embeddings Capture Scales?](https://mlsanity.com/paper/2010.05345/) (2020)
- [Broaden the Vision: Geo-Diverse Visual Commonsense Reasoning](https://mlsanity.com/paper/2109.06860/) (2021)
- [Welcome to the Modern World of Pronouns: Identity-Inclusive Natural Language Processing beyond Gender](https://mlsanity.com/paper/2202.11923/) (2022)
- [Things not Written in Text: Exploring Spatial Commonsense from Visual Signals](https://mlsanity.com/paper/2203.08075/) (2022)
- [Towards an Atlas of Cultural Commonsense for Machine Reasoning](https://mlsanity.com/paper/2009.05664/) (2020)

## Built on by (most cited first, 30 of 30)

- [Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models](https://mlsanity.com/paper/2310.10378/) (2023)
- [CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologies](https://mlsanity.com/paper/2404.15238/) (2024)
- [Extracting Cultural Commonsense Knowledge at Scale](https://mlsanity.com/paper/2210.07763/) (2022)
- [GPT4GEO: How a Language Model Sees the World's Geography](https://mlsanity.com/paper/2306.00020/) (2023)
- [CULTURE-GEN: Revealing Global Cultural Perception in Language Models through Natural Language Prompting](https://mlsanity.com/paper/2404.10199/) (2024)
- [Knowledge of cultural moral norms in large language models](https://mlsanity.com/paper/2306.01857/) (2023)
- [Inspecting the Geographical Representativeness of Images from Text-to-Image Models](https://mlsanity.com/paper/2305.11080/) (2023)
- [Benchmarks for Automated Commonsense Reasoning: A Survey](https://mlsanity.com/paper/2302.04752/) (2023)
- [IndoCulture: Exploring Geographically-Influenced Cultural Commonsense Reasoning Across Eleven Indonesian Provinces](https://mlsanity.com/paper/2404.01854/) (2024)
- [Speaking Multiple Languages Affects the Moral Bias of Language Models](https://mlsanity.com/paper/2211.07733/) (2022)
- [Geographic and Geopolitical Biases of Language Models](https://mlsanity.com/paper/2212.10408/) (2022)
- [Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Art](https://mlsanity.com/paper/2406.03930/) (2024)
- [GradientCoin: A Peer-to-Peer Decentralized Large Language Models](https://mlsanity.com/paper/2308.10502/) (2023)
- [Cultural Commonsense Knowledge for Intercultural Dialogues](https://mlsanity.com/paper/2402.10689/) (2024)
- [Localized Cultural Knowledge is Conserved and Controllable in Large Language Models](https://mlsanity.com/paper/2504.10191/) (2025)
- [Understanding Finetuning for Factual Knowledge Extraction from Language Models](https://mlsanity.com/paper/2301.11293/) (2023)
- [TaskLAMA: Probing the Complex Task Understanding of Language Models](https://mlsanity.com/paper/2308.15299/) (2023)
- [GIVL: Improving Geographical Inclusivity of Vision-Language Models with Pre-Training Methods](https://mlsanity.com/paper/2301.01893/) (2023)
- [On the Consistency of Multilingual Context Utilization in Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.00597/) (2025)
- [How Well Do LLMs Identify Cultural Unity in Diversity?](https://mlsanity.com/paper/2408.05102/) (2024)
- [SaudiCulture: A Benchmark for Evaluating Large Language Models Cultural Competence within Saudi Arabia](https://mlsanity.com/paper/2503.17485/) (2025)
- [Temporal Blind Spots in Large Language Models](https://mlsanity.com/paper/2401.12078/) (2024)
- [Characterizing Bias: Benchmarking Large Language Models in Simplified versus Traditional Chinese](https://mlsanity.com/paper/2505.22645/) (2025)
- [The World According to LLMs: How Geographic Origin Influences LLMs' Entity Deduction Capabilities](https://mlsanity.com/paper/2508.05525/) (2025)
- [VIPHY: Probing "Visible" Physical Commonsense Knowledge](https://mlsanity.com/paper/2209.07000/) (2022)
- [Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs](https://mlsanity.com/paper/2604.19292/) (2026)
- [Camellia: Benchmarking Cultural Biases in LLMs for Asian Languages](https://mlsanity.com/paper/2510.05291/) (2025)
- [Matching Tasks to Objectives: Fine-Tuning and Prompt-Tuning Strategies for Encoder-Decoder Pre-trained Language Models](https://mlsanity.com/paper/2606.24841/) (2026)
- [Is a Prestigious Job the same as a Prestigious Country? A Case Study on Multilingual Sentence Embeddings and European Countries](https://mlsanity.com/paper/2305.14482/) (2023)
- [Tahakom LLM Guidelines and Recipes: From Pre-training Data to an Arabic LLM](https://mlsanity.com/paper/2510.13481/) (2025)

## Abstract (arXiv)

Recent work has shown that Pre-trained Language Models (PLMs) store the relational knowledge learned from data and utilize it for performing downstream tasks. However, commonsense knowledge across different regions may vary. For instance, the color of bridal dress is white in American weddings whereas it is red in Chinese weddings. In this paper, we introduce a benchmark dataset, Geo-Diverse Commonsense Multilingual Language Models Analysis (GeoMLAMA), for probing the diversity of the relational knowledge in multilingual PLMs. GeoMLAMA contains 3,125 prompts in English, Chinese, Hindi, Persian, and Swahili, with a wide coverage of concepts shared by people from American, Chinese, Indian, Iranian and Kenyan cultures. We benchmark 11 standard multilingual PLMs on GeoMLAMA. Interestingly, we find that 1) larger multilingual PLMs variants do not necessarily store geo-diverse concepts better than its smaller variant; 2) multilingual PLMs are not intrinsically biased towards knowledge from the Western countries (the United States); 3) the native language of a country may not be the best language to probe its knowledge and 4) a language may better probe knowledge about a non-native country than its native country. Code and data are released at https://github.com/WadeYin9712/GeoMLAMA.