# Pre-Trained Language Models Represent Some Geographic Populations Better Than Others

arXiv: https://arxiv.org/abs/2403.11025 · HTML: https://mlsanity.com/paper/2403.11025/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 29 references that are in the map.
- First posted on arXiv 16 Mar 2024.

## Builds on (most cited first, 18 of 18)

- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Are All Languages Created Equal in Multilingual BERT?](https://mlsanity.com/paper/2005.09093/) (2020)
- [Mapping Languages: The Corpus of Global Language Use](https://mlsanity.com/paper/2004.00798/) (2020)
- [When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfer](https://mlsanity.com/paper/2110.14782/) (2021)
- [Detecting Cross-Geographic Biases in Toxicity Modeling on Social Media](https://mlsanity.com/paper/2104.06999/) (2021)
- [Counterfactual reasoning: Testing language models' understanding of hypothetical scenarios](https://mlsanity.com/paper/2305.16572/) (2023)
- [Geographic Citation Gaps in NLP Research](https://mlsanity.com/paper/2210.14424/) (2022)
- [Geographic Adaptation of Pretrained Language Models](https://mlsanity.com/paper/2203.08565/) (2022)
- [Representations of Language Varieties Are Reliable Given Corpus Similarity Measures](https://mlsanity.com/paper/2104.01294/) (2021)
- [Confounds and Consequences in Geotagged Twitter Data](https://mlsanity.com/paper/1506.02275/) (2015)
- [Geographical Distance Is The New Hyperparameter: A Case Study Of Finding The Optimal Pre-trained Language For English-isiZulu Machine Translation](https://mlsanity.com/paper/2205.08621/) (2022)
- [Measuring Geographic Performance Disparities of Offensive Language Classifiers](https://mlsanity.com/paper/2209.07353/) (2022)
- [Corpus Similarity Measures Remain Robust Across Diverse Languages](https://mlsanity.com/paper/2206.04332/) (2022)
- [Register Variation Remains Stable Across 60 Languages](https://mlsanity.com/paper/2209.09813/) (2022)
- [cantnlp@LT-EDI-2023: Homophobia/Transphobia Detection in Social Media Comments using Spatio-Temporally Retrained Language Models](https://mlsanity.com/paper/2308.10370/) (2023)

## Built on by (most cited first, 6 of 6)

- [Language Model Alignment in Multilingual Trolley Problems](https://mlsanity.com/paper/2407.02273/) (2024)
- [From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centered](https://mlsanity.com/paper/2506.07461/) (2025)
- [XCR-Bench: Benchmarking Cross-Cultural Reasoning in LLMs via Culture-Specific Items and Hall's Triad](https://mlsanity.com/paper/2601.14063/) (2026)
- [Uncovering Regional Defaults from Photorealistic Forests in Text-to-Image Generation with DALL-E 2](https://mlsanity.com/paper/2410.17255/) (2024)
- [Performance Gains of LLMs With Humans in a World of LLMs Versus Humans](https://mlsanity.com/paper/2505.08902/) (2025)
- [Mapping the City Through the Lens of Language Models](https://mlsanity.com/paper/2608.02971/) (2026)

## Abstract (arXiv)

This paper measures the skew in how well two families of LLMs represent diverse geographic populations. A spatial probing task is used with geo-referenced corpora to measure the degree to which pre-trained language models from the OPT and BLOOM series represent diverse populations around the world. Results show that these models perform much better for some populations than others. In particular, populations across the US and the UK are represented quite well while those in South and Southeast Asia are poorly represented. Analysis shows that both families of models largely share the same skew across populations. At the same time, this skew cannot be fully explained by sociolinguistic factors, economic factors, or geographic factors. The basic conclusion from this analysis is that pre-trained models do not equally represent the world's population: there is a strong skew towards specific geographic populations. This finding challenges the idea that a single model can be used for all populations.