# The World According to LLMs: How Geographic Origin Influences LLMs' Entity Deduction Capabilities

arXiv: https://arxiv.org/abs/2508.05525 · HTML: https://mlsanity.com/paper/2508.05525/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 44 references that are in the map.
- First posted on arXiv 7 Aug 2025.

## Builds on (most cited first, 30 of 33)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://mlsanity.com/paper/2010.00133/) (2020)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research](https://mlsanity.com/paper/2402.00159/) (2024)
- [Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models](https://mlsanity.com/paper/2401.06102/) (2024)
- [Asking Clarifying Questions in Open-Domain Information-Seeking Conversations](https://mlsanity.com/paper/1907.06554/) (2019)
- [What's In My Big Data?](https://mlsanity.com/paper/2310.20707/) (2023)
- [GuessWhat?! Visual object discovery through multi-modal dialogue](https://mlsanity.com/paper/1611.08481/) (2016)
- [Large Language Models are Geographically Biased](https://mlsanity.com/paper/2402.02680/) (2024)
- [Mitigating Language-Dependent Ethnic Bias in BERT](https://mlsanity.com/paper/2109.05704/) (2021)
- [Toward Gender-Inclusive Coreference Resolution](https://mlsanity.com/paper/1910.13913/) (2019)
- [Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned Language Models](https://mlsanity.com/paper/2101.09688/) (2021)
- [Measuring Implicit Bias in Explicitly Unbiased Large Language Models](https://mlsanity.com/paper/2402.04105/) (2024)
- [Towards Implicit Bias Detection and Mitigation in Multi-Agent LLM Interactions](https://mlsanity.com/paper/2410.02584/) (2024)
- [Disclosure and Mitigation of Gender Bias in LLMs](https://mlsanity.com/paper/2402.11190/) (2024)
- [A survey of bias in Machine Learning through the prism of Statistical Parity for the Adult Data Set](https://mlsanity.com/paper/2003.14263/) (2020)
- [Unmasking and Quantifying Racial Bias of Large Language Models in Medical Report Generation](https://mlsanity.com/paper/2401.13867/) (2024)
- [GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Models](https://mlsanity.com/paper/2205.12247/) (2022)
- [Sociodemographic Bias in Language Models: A Survey and Forward Path](https://mlsanity.com/paper/2306.08158/) (2023)
- [Explicit vs. Implicit: Investigating Social Bias in Large Language Models through Self-Reflection](https://mlsanity.com/paper/2501.02295/) (2025)
- [Investigating Implicit Bias in Large Language Models: A Large-Scale Study of Over 50 LLMs](https://mlsanity.com/paper/2410.12864/) (2024)
- [Probing the Multi-turn Planning Capabilities of LLMs via 20 Question Games](https://mlsanity.com/paper/2310.01468/) (2023)
- [ChatGPT MT: Competitive for High- (but not Low-) Resource Languages](https://mlsanity.com/paper/2309.07423/) (2023)
- [Learning to Ask Informative Questions: Enhancing LLMs with Preference Optimization and Expected Information Gain](https://mlsanity.com/paper/2406.17453/) (2024)
- [Multi-FAct: Assessing Factuality of Multilingual LLMs using FActScore](https://mlsanity.com/paper/2402.18045/) (2024)
- [Contrastive Multi-document Question Generation](https://mlsanity.com/paper/1911.03047/) (2019)
- [Geographical Erasure in Language Generation](https://mlsanity.com/paper/2310.14777/) (2023)
- [Tackling Bias in Pre-trained Language Models: Current Trends and Under-represented Societies](https://mlsanity.com/paper/2312.01509/) (2023)

## Built on by (most cited first, 5 of 5)

- [Calibrate-Then-Act: Cost-Aware Exploration in LLM Agents](https://mlsanity.com/paper/2602.16699/) (2026)
- [Auditing Preferences for Brands and Cultures in LLMs](https://mlsanity.com/paper/2603.18300/) (2026)
- [Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs](https://mlsanity.com/paper/2604.19292/) (2026)
- [Are Video Generation Models Geographically Fair? An Attraction-Centric Evaluation of Global Visual Knowledge](https://mlsanity.com/paper/2601.18698/) (2026)
- [Benchmarking Open-Weight Foundation Models for Global AI Technical Governance](https://mlsanity.com/paper/2606.26099/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have been extensively tuned to mitigate explicit biases, yet they often exhibit subtle implicit biases rooted in their pre-training data. Rather than directly probing LLMs with human-crafted questions that may trigger guardrails, we propose studying how models behave when they proactively ask questions themselves. The 20 Questions game, a multi-turn deduction task, serves as an ideal testbed for this purpose. We systematically evaluate geographic performance disparities in entity deduction using a new dataset, Geo20Q+, consisting of both notable people and culturally significant objects (e.g., foods, landmarks, animals) from diverse regions. We test popular LLMs across two gameplay configurations (canonical 20-question and unlimited turns) and in seven languages (English, Hindi, Mandarin, Japanese, French, Spanish, and Turkish). Our results reveal geographic disparities: LLMs are substantially more successful at deducing entities from the Global North than the Global South, and the Global West than the Global East. While Wikipedia pageviews and pre-training corpus frequency correlate mildly with performance, they fail to fully explain these disparities. Notably, the language in which the game is played has minimal impact on performance gaps. These findings demonstrate the value of creative, free-form evaluation frameworks for uncovering subtle biases in LLMs that remain hidden in standard prompting setups. By analyzing how models initiate and pursue reasoning goals over multiple turns, we find geographic and cultural disparities embedded in their reasoning processes. We release the dataset (Geo20Q+) and code at https://sites.google.com/view/llmbias20q/home.