# A Dual-Layered Evaluation of Geopolitical and Cultural Bias in LLMs

arXiv: https://arxiv.org/abs/2506.21881 · HTML: https://mlsanity.com/paper/2506.21881/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 22 references that are in the map.
- First posted on arXiv 27 Jun 2025.

## Builds on (most cited first, 12 of 12)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Persistent Anti-Muslim Bias in Large Language Models](https://mlsanity.com/paper/2101.05783/) (2021)
- [From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models](https://mlsanity.com/paper/2305.08283/) (2023)
- [Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study](https://mlsanity.com/paper/2303.17466/) (2023)
- [Towards Measuring and Modeling "Culture" in LLMs: A Survey](https://mlsanity.com/paper/2403.15412/) (2024)
- [Probing Pre-Trained Language Models for Cross-Cultural Differences in Values](https://mlsanity.com/paper/2203.13722/) (2022)
- [Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models](https://mlsanity.com/paper/2310.10378/) (2023)
- [This Land is {Your, My} Land: Evaluating Geopolitical Biases in Language Models](https://mlsanity.com/paper/2305.14610/) (2023)
- [Optimizing Language Augmentation for Multilingual Large Language Models: A Case Study on Korean](https://mlsanity.com/paper/2403.10882/) (2024)
- [RakutenAI-7B: Extending Large Language Models for Japanese](https://mlsanity.com/paper/2403.15484/) (2024)

## Built on by (most cited first, 5 of 5)

- [Mind the Gap in Cultural Alignment: Task-Aware Culture Management for Large Language Models](https://mlsanity.com/paper/2602.22475/) (2026)
- [Structuring the Space of Sociotechnical Alignment](https://mlsanity.com/paper/2607.01250/) (2026)
- [FIBER: A Multilingual Evaluation Resource for Factual Inference Bias](https://mlsanity.com/paper/2512.11110/) (2025)
- [We're Cooked! - Probing LLM Political Alignment Via Conflict-Framed Recipe Translation](https://mlsanity.com/paper/2609.07568/) (2026)
- [CultureMINE: Datasets and Methods for Improving the Cultural Capabilities of NLP Systems](https://mlsanity.com/paper/2609.22494/) (2026)

## Abstract (arXiv)

As large language models (LLMs) are increasingly deployed across diverse linguistic and cultural contexts, understanding their behavior in both factual and disputable scenarios is essential, especially when their outputs may shape public opinion or reinforce dominant narratives. In this paper, we define two types of bias in LLMs: model bias (bias stemming from model training) and inference bias (bias induced by the language of the query), through a two-phase evaluation. Phase 1 evaluates LLMs on factual questions where a single verifiable answer exists, assessing whether models maintain consistency across different query languages. Phase 2 expands the scope by probing geopolitically sensitive disputes, where responses may reflect culturally embedded or ideologically aligned perspectives. We construct a manually curated dataset spanning both factual and disputable QA, across four languages and question types. The results show that Phase 1 exhibits query language induced alignment, while Phase 2 reflects an interplay between the model's training context and query language. This paper offers a structured framework for evaluating LLM behavior across neutral and sensitive topics, providing insights for future LLM deployment and culturally aware evaluation practices in multilingual contexts.