# Assessing the Political Fairness of Multilingual LLMs: A Case Study based on a 21-way Multiparallel EuroParl Dataset

arXiv: https://arxiv.org/abs/2510.20508 · HTML: https://mlsanity.com/paper/2510.20508/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 86 references that are in the map.
- First posted on arXiv 23 Oct 2025.

## Builds on (most cited first, 30 of 31)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [Multimodal datasets: misogyny, pornography, and malignant stereotypes](https://mlsanity.com/paper/2110.01963/) (2021)
- [From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models](https://mlsanity.com/paper/2305.08283/) (2023)
- [Tower: An Open Multilingual Large Language Model for Translation-Related Tasks](https://mlsanity.com/paper/2402.17733/) (2024)
- [The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation](https://mlsanity.com/paper/2301.01768/) (2023)
- [Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models](https://mlsanity.com/paper/2402.16786/) (2024)
- [Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debates](https://mlsanity.com/paper/1911.03167/) (2019)
- [EuroLLM: Multilingual Language Models for Europe](https://mlsanity.com/paper/2409.16235/) (2024)
- [Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation](https://mlsanity.com/paper/1808.07048/) (2018)
- [Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics](https://mlsanity.com/paper/2106.14574/) (2021)
- [A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methods](https://mlsanity.com/paper/2104.00054/) (2021)
- [Hidden Persuaders: LLMs' Political Leaning and Their Influence on Voters](https://mlsanity.com/paper/2410.24190/) (2024)
- [Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processing](https://mlsanity.com/paper/2101.03289/) (2021)
- [Beyond prompt brittleness: Evaluating the reliability and consistency of political worldviews in LLMs](https://mlsanity.com/paper/2402.17649/) (2024)
- [The Effect of Translationese in Machine Translation Test Sets](https://mlsanity.com/paper/1906.08069/) (2019)
- [Opportunities and Risks of LLMs for Scalable Deliberation with Polis](https://mlsanity.com/paper/2306.11932/) (2023)
- [Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies](https://mlsanity.com/paper/2401.06760/) (2024)
- [It's Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Information](https://mlsanity.com/paper/2005.02354/) (2020)
- [IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistance](https://mlsanity.com/paper/2502.08395/) (2025)
- [Pitfalls and Outlooks in Using COMET](https://mlsanity.com/paper/2408.15366/) (2024)
- [What Is The Political Content in LLMs' Pre- and Post-Training Data?](https://mlsanity.com/paper/2509.22367/) (2025)
- [Multi-EuP: The Multilingual European Parliament Dataset for Analysis of Bias in Information Retrieval](https://mlsanity.com/paper/2311.01870/) (2023)

## Abstract (arXiv)

The political biases of Large Language Models (LLMs) are usually assessed by simulating their answers to English surveys. In this work, we propose an alternative framing of political biases, relying on principles of fairness in multilingual translation. We systematically compare the translation quality of speeches in the European Parliament (EP), observing systematic differences with majority parties from left and right being better translated than outsider parties. This study is made possible by a new, 21-way multiparallel version of EuroParl, the parliamentary proceedings of the EP, which includes the political affiliations of each speaker. The dataset consists of 1.5M sentences for a total of 40M words and 249M characters. It covers three years, 1000+ speakers, 7 countries, 12 EU parties, 25 EU committees, and hundreds of national parties.