# Assessing Agentic Large Language Models in Multilingual National Bias

arXiv: https://arxiv.org/abs/2502.17945 · HTML: https://mlsanity.com/paper/2502.17945/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 20 of its 25 references that are in the map.
- First posted on arXiv 25 Feb 2025.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 20 of 20)

- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [Having Beer after Prayer? Measuring Cultural Bias in Large Language Models](https://mlsanity.com/paper/2305.14456/) (2023)
- [Systematic Inequalities in Language Technology Performance across the World's Languages](https://mlsanity.com/paper/2110.06733/) (2021)
- [Language Model Alignment in Multilingual Trolley Problems](https://mlsanity.com/paper/2407.02273/) (2024)
- [Evaluating Cultural Adaptability of a Large Language Model via Simulation of Synthetic Personas](https://mlsanity.com/paper/2408.06929/) (2024)
- [Gender Bias in Large Language Models across Multiple Languages](https://mlsanity.com/paper/2403.00277/) (2024)
- [MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMs](https://mlsanity.com/paper/2406.07243/) (2024)
- [This Land is {Your, My} Land: Evaluating Geopolitical Biases in Language Models](https://mlsanity.com/paper/2305.14610/) (2023)
- [Investigating Subtler Biases in LLMs: Ageism, Beauty, Institutional, and Nationality Bias in Generative Models](https://mlsanity.com/paper/2309.08902/) (2023)
- [On Evaluating and Mitigating Gender Biases in Multilingual Settings](https://mlsanity.com/paper/2307.01503/) (2023)
- [The Silicon Ceiling: Auditing GPT's Race and Gender Biases in Hiring](https://mlsanity.com/paper/2405.04412/) (2024)
- [Global Voices, Local Biases: Socio-Cultural Prejudices across Languages](https://mlsanity.com/paper/2310.17586/) (2023)
- [Language Representation Projection: Can We Transfer Factual Knowledge across Languages in Multilingual Language Models?](https://mlsanity.com/paper/2311.03788/) (2023)
- [Quite Good, but Not Enough: Nationality Bias in Large Language Models -- A Case Study of ChatGPT](https://mlsanity.com/paper/2405.06996/) (2024)
- [Do Multilingual Large Language Models Mitigate Stereotype Bias?](https://mlsanity.com/paper/2407.05740/) (2024)
- [What an Elegant Bridge: Multilingual LLMs are Biased Similarly in Different Languages](https://mlsanity.com/paper/2407.09704/) (2024)
- [Dissecting Bias of ChatGPT in College Major Recommendations](https://mlsanity.com/paper/2401.11699/) (2023)

## Abstract (arXiv)

Large Language Models have garnered significant attention for their capabilities in multilingual natural language processing, while studies on risks associated with cross biases are limited to immediate context preferences. Cross-language disparities in reasoning-based recommendations remain largely unexplored, with a lack of even descriptive analysis. This study is the first to address this gap. We test LLM's applicability and capability in providing personalized advice across three key scenarios: university applications, travel, and relocation. We investigate multilingual bias in state-of-the-art LLMs by analyzing their responses to decision-making tasks across multiple languages. We quantify bias in model-generated scores and assess the impact of demographic factors and reasoning strategies (e.g., Chain-of-Thought prompting) on bias patterns. Our findings reveal that local language bias is prevalent across different tasks, with GPT-4 and Sonnet reducing bias for English-speaking countries compared to GPT-3.5 but failing to achieve robust multilingual alignment, highlighting broader implications for multilingual AI agents and applications such as education. footnoteCode available at: https://github.com/yiyunya/assessagenticnationalbias