mlsanityData to 7 Oct 2026

Assessing the Political Fairness of Multilingual LLMs: A Case Study based on a 21-way Multiparallel EuroParl Dataset

arXiv 2510.20508 ↗First posted 23 Oct 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

The political biases of Large Language Models (LLMs) are usually assessed by simulating their answers to English surveys. In this work, we propose an alternative framing of political biases, relying on principles of fairness in multilingual translation. We systematically compare the translation quality of speeches in the European Parliament (EP), observing systematic differences with majority parties from left and right being better translated than outsider parties. This study is made possible by a new, 21-way multiparallel version of EuroParl, the parliamentary proceedings of the EP, which includes the political affiliations of each speaker. The dataset consists of 1.5M sentences for a total of 40M words and 249M characters. It covers three years, 1000+ speakers, 7 countries, 12 EU parties, 25 EU committees, and hundreds of national parties.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 31 of its 86 references that are in the map. First posted on arXiv 23 Oct 2025.

Builds on

31 of 86 references.

References · 31

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  6. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  7. 2023Whose Opinions Do Language Models Reflect?cited by 759
  8. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  9. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  10. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 326
  11. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 257
  12. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 205
19 more
  1. 2023The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientationcited by 183
  2. 2024Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Modelscited by 162
  3. 2019Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debatescited by 114
  4. 2024EuroLLM: Multilingual Language Models for Europecited by 103
  5. 2018Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluationcited by 88
  6. 2021Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metricscited by 83
  7. 2021A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methodscited by 71
  8. 2024Hidden Persuaders: LLMs' Political Leaning and Their Influence on Voterscited by 71
  9. 2021Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processingcited by 68
  10. 2024Beyond prompt brittleness: Evaluating the reliability and consistency of political worldviews in LLMscited by 55
  11. 2019The Effect of Translationese in Machine Translation Test Setscited by 49
  12. 2023Opportunities and Risks of LLMs for Scalable Deliberation with Poliscited by 41
  13. 2024Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuraciescited by 40
  14. 2020It's Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Informationcited by 26
  15. 2025IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistancecited by 22
  16. 2024Pitfalls and Outlooks in Using COMETcited by 10
  17. 2025What Is The Political Content in LLMs' Pre- and Post-Training Data?cited by 7
  18. 2023Multi-EuP: The Multilingual European Parliament Dataset for Analysis of Bias in Information Retrievalcited by 4
  19. 2022Modeling Target-Side Morphology in Neural Machine Translation: A Comparison of Strategiescited by 1

Built on by

Not cited yet.