# This Land is {Your, My} Land: Evaluating Geopolitical Biases in Language Models

arXiv: https://arxiv.org/abs/2305.14610 · HTML: https://mlsanity.com/paper/2305.14610/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 28 references that are in the map.
- First posted on arXiv 24 May 2023.

## Builds on (most cited first, 15 of 15)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://mlsanity.com/paper/2010.00133/) (2020)
- [MEGA: Multilingual Evaluation of Generative AI](https://mlsanity.com/paper/2303.12528/) (2023)
- [From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models](https://mlsanity.com/paper/2305.08283/) (2023)
- [Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study](https://mlsanity.com/paper/2303.17466/) (2023)
- [Probing Pre-Trained Language Models for Cross-Cultural Differences in Values](https://mlsanity.com/paper/2203.13722/) (2022)
- [Bidirectional Language Models Are Also Few-shot Learners](https://mlsanity.com/paper/2209.14500/) (2022)
- [On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codex](https://mlsanity.com/paper/2301.12868/) (2023)
- [Geographic and Geopolitical Biases of Language Models](https://mlsanity.com/paper/2212.10408/) (2022)
- [Bloom Library: Multimodal Datasets in 300+ Languages for a Variety of Downstream Tasks](https://mlsanity.com/paper/2210.14712/) (2022)

## Built on by (most cited first, 30 of 32)

- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [Investigating Cultural Alignment of Large Language Models](https://mlsanity.com/paper/2402.13231/) (2024)
- [Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above](https://mlsanity.com/paper/2502.14127/) (2025)
- [Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignment](https://mlsanity.com/paper/2404.01054/) (2024)
- [Multilingual Retrieval Augmented Generation for Culturally-Sensitive Tasks: A Benchmark for Cross-lingual Robustness](https://mlsanity.com/paper/2410.01171/) (2024)
- [Beyond Words: Exploring Cultural Value Sensitivity in Multimodal Models](https://mlsanity.com/paper/2502.14906/) (2025)
- [Vox Populi, Vox AI? Using Language Models to Estimate German Public Opinion](https://mlsanity.com/paper/2407.08563/) (2024)
- [Eliciting Better Multilingual Structured Reasoning from LLMs through Code](https://mlsanity.com/paper/2403.02567/) (2024)
- [A Reality Check on Context Utilisation for Retrieval-Augmented Generation](https://mlsanity.com/paper/2412.17031/) (2024)
- [Faux Polyglot: A Study on Information Disparity in Multilingual Large Language Models](https://mlsanity.com/paper/2407.05502/) (2024)
- [Geopolitical biases in LLMs: what are the "good" and the "bad" countries according to contemporary language models](https://mlsanity.com/paper/2506.06751/) (2025)
- [Whose Truth? Pluralistic Geo-Alignment for (Agentic) AI](https://mlsanity.com/paper/2508.05432/) (2025)
- [United in Diversity? Contextual Biases in LLM-Based Predictions of the 2024 European Parliament Elections](https://mlsanity.com/paper/2409.09045/) (2024)
- [Echoes of Power: Investigating Geopolitical Bias in US and China Large Language Models](https://mlsanity.com/paper/2503.16679/) (2025)
- [Improving Multilingual Retrieval-Augmented Language Models through Dialectic Reasoning Argumentations](https://mlsanity.com/paper/2504.04771/) (2025)
- [A Dual-Layered Evaluation of Geopolitical and Cultural Bias in LLMs](https://mlsanity.com/paper/2506.21881/) (2025)
- [Selected Languages are All You Need for Cross-lingual Truthfulness Transfer](https://mlsanity.com/paper/2406.14434/) (2024)
- [Uncovering Differences in Persuasive Language in Russian versus English Wikipedia](https://mlsanity.com/paper/2409.19148/) (2024)
- [An Independent Safety Evaluation of Kimi K2.5](https://mlsanity.com/paper/2604.03121/) (2026)
- [Conflicts in Texts: Data, Implications and Challenges](https://mlsanity.com/paper/2504.19472/) (2025)
- [Language Shapes Mental Health Evaluations in Large Language Models](https://mlsanity.com/paper/2603.06910/) (2026)
- [Implicit Values Embedded in How Humans and LLMs Complete Subjective Everyday Tasks](https://mlsanity.com/paper/2510.03384/) (2025)
- [Assessing Agentic Large Language Models in Multilingual National Bias](https://mlsanity.com/paper/2502.17945/) (2025)
- [Controlled Territory and Conflict Tracking (CONTACT): (Geo-)Mapping Occupied Territory from Open Source Intelligence](https://mlsanity.com/paper/2504.13730/) (2025)
- [Coordinates from Context: Using LLMs to Ground Complex Location References](https://mlsanity.com/paper/2510.08741/) (2025)
- [FIBER: A Multilingual Evaluation Resource for Factual Inference Bias](https://mlsanity.com/paper/2512.11110/) (2025)
- [Analysing LLM Persona Generation and Fairness Interpretation in Polarised Geopolitical Contexts](https://mlsanity.com/paper/2603.22837/) (2026)
- [The Shibboleth Effect: Auditing the Cross-Lingual Distributional Skew of Large Language Models](https://mlsanity.com/paper/2606.11082/) (2026)
- [Evaluating Regional Bias in LLMs From Abstract Stereotype to Concrete Social Decision-Making](https://mlsanity.com/paper/2607.27022/) (2026)
- [Mapping the City Through the Lens of Language Models](https://mlsanity.com/paper/2608.02971/) (2026)

## Abstract (arXiv)

Do the Spratly Islands belong to China, the Philippines, or Vietnam? A pretrained large language model (LLM) may answer differently if asked in the languages of each claimant country: Chinese, Tagalog, or Vietnamese. This contrasts with a multilingual human, who would likely answer consistently. In this paper, we show that LLMs recall certain geographical knowledge inconsistently when queried in different languages -- a phenomenon we term geopolitical bias. As a targeted case study, we consider territorial disputes, an inherently controversial and multilingual task. We introduce BorderLines, a dataset of territorial disputes which covers 251 territories, each associated with a set of multiple-choice questions in the languages of each claimant country (49 languages in total). We also propose a suite of evaluation metrics to precisely quantify bias and consistency in responses across different languages. We then evaluate various multilingual LLMs on our dataset and metrics to probe their internal knowledge and use the proposed metrics to discover numerous inconsistencies in how these models respond in different languages. Finally, we explore several prompt modification strategies, aiming to either amplify or mitigate geopolitical bias, which highlights how brittle LLMs are and how they tailor their responses depending on cues from the interaction context. Our code and data are available at https://github.com/manestay/borderlines