# Ready to Translate, Not to Represent? Bias and Performance Gaps in Multilingual LLMs Across Language Families and Domains

arXiv: https://arxiv.org/abs/2510.07877 · HTML: https://mlsanity.com/paper/2510.07877/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 39 references that are in the map.
- First posted on arXiv 9 Oct 2025.

## Builds on (most cited first, 19 of 19)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [Social Bias Frames: Reasoning about Social and Power Implications of Language](https://mlsanity.com/paper/1911.03891/) (2019)
- [Six Challenges for Neural Machine Translation](https://mlsanity.com/paper/1706.03872/) (2017)
- [How do Large Language Models Handle Multilingualism?](https://mlsanity.com/paper/2402.18815/) (2024)
- [MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer](https://mlsanity.com/paper/2109.00904/) (2021)
- [A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias](https://mlsanity.com/paper/2404.00929/) (2024)
- [Not Low-Resource Anymore: Aligner Ensembling, Batch Filtering, and New Datasets for Bengali-English Machine Translation](https://mlsanity.com/paper/2009.09359/) (2020)
- [Evaluating o1-Like LLMs: Unlocking Reasoning for Translation through Comprehensive Analysis](https://mlsanity.com/paper/2502.11544/) (2025)
- [MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languages](https://mlsanity.com/paper/2504.10356/) (2025)
- [GPT-4 vs. Human Translators: A Comprehensive Evaluation of Translation Quality Across Languages, Domains, and Expertise Levels](https://mlsanity.com/paper/2407.03658/) (2024)
- [The power of Prompts: Evaluating and Mitigating Gender Bias in MT with LLMs](https://mlsanity.com/paper/2407.18786/) (2024)
- [Large Language Model for Multi-Domain Translation: Benchmarking and Domain CoT Fine-tuning](https://mlsanity.com/paper/2410.02631/) (2024)
- [LLM-Assisted Rule Based Machine Translation for Low/No-Resource Languages](https://mlsanity.com/paper/2405.08997/) (2024)
- [Improving Character Error Rate Is Not Equal to Having Clean Speech: Speech Enhancement for ASR Systems with Black-box Acoustic Models](https://mlsanity.com/paper/2110.05968/) (2021)

## Built on by (most cited first, 1 of 1)

- [The Language Blind Spot: How Query Language and Brand Recognition Tier Shape AI-Constructed Brand Reputation Across Twelve European Languages](https://mlsanity.com/paper/2606.23165/) (2026)

## Abstract (arXiv)

The rise of Large Language Models (LLMs) has redefined Machine Translation (MT), enabling context-aware and fluent translations across hundreds of languages and textual domains. Despite their remarkable capabilities, LLMs often exhibit uneven performance across language families and specialized domains. Moreover, recent evidence reveals that these models can encode and amplify different biases present in their training data, posing serious concerns for fairness, especially in low-resource languages. To address these gaps, we introduce Translation Tangles, a unified framework and dataset for evaluating the translation quality and fairness of open-source LLMs. Our approach benchmarks 24 bidirectional language pairs across multiple domains using different metrics. We further propose a hybrid bias detection pipeline that integrates rule-based heuristics, semantic similarity filtering, and LLM-based validation. We also introduce a high-quality, bias-annotated dataset based on human evaluations of 1,439 translation-reference pairs. The code and dataset are accessible on GitHub: https://github.com/faiyazabdullah/TranslationTangles