mlsanityData to 14 Oct 2026

Evaluating Machine Translation Datasets for Low-Web Data Languages: A Gendered Lens

arXiv 2511.03880 ↗First posted 5 Nov 2025cs.CL cs.CYcited by 0

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

As low-resourced languages are increasingly incorporated into NLP research, there is an emphasis on collecting large-scale datasets. But in prioritizing quantity over quality, we risk 1) building language technologies that perform poorly for these languages and 2) producing harmful content that perpetuates societal biases. In this paper, we investigate the quality of Machine Translation (MT) datasets for three low-resourced languages--Afan Oromo, Amharic, and Tigrinya, with a focus on the gender representation in the datasets. Our findings demonstrate that while training data has a large representation of political and religious domain text, benchmark datasets are focused on news, health, and sports. We also found a large skew towards the male gender--in names of persons, the grammatical gender of verbs, and in stereotypical depictions in the datasets. Further, we found harmful and toxic depictions against women, which were more prominent for the language with the largest amount of data, underscoring that quantity does not guarantee quality. We hope that our work inspires further inquiry into the datasets collected for low-resourced languages and prompts early mitigation of harmful content. WARNING: This paper contains discussion of NSFW content that some may find disturbing.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 27 of its 73 references that are in the map. First posted on arXiv 5 Nov 2025.

Builds on

27 of 73 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  3. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,067
  4. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  5. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  6. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 659
  7. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  8. 2019A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cyclecited by 193
  9. 2024Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluationcited by 191
  10. 2021MasakhaNER: Named Entity Recognition for African Languagescited by 155
  11. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  12. 2021Gender Bias in Machine Translationcited by 117
15 more
  1. 2022A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translationcited by 98
  2. 2022AfroLM: A Self-Active Learning-based Multilingual Pretrained Language Model for 23 African Languagescited by 46
  3. 2022Handling and Presenting Harmful Text in NLP Researchcited by 43
  4. 2023Making More of Little Data: Improving Low-Resource Automatic Speech Recognition Using Data Augmentationcited by 31
  5. 2023On Hate Scaling Laws For Data-Swampscited by 28
  6. 2024LLMs are Biased Teachers: Evaluating LLM Bias in Personalized Educationcited by 28
  7. 2018SemAxis: A Lightweight Framework to Characterize Domain-Specific Word Semantics Beyond Sentimentcited by 26
  8. 2024Adapting Multilingual LLMs to Low-Resource Languages using Continued Pre-training and Synthetic Corpuscited by 24
  9. 2023A Material Lens on Coloniality in NLPcited by 21
  10. 2024Akal Badi ya Bias: An Exploratory Study of Gender Bias in Hindi Language Technologycited by 17
  11. 2024What the Harm? Quantifying the Tangible Impact of Gender Bias in Machine Translation with a Human-centered Studycited by 15
  12. 2021Extended Parallel Corpus for Amharic-English Machine Translationcited by 10
  13. 2021Crowdsourcing Parallel Corpus for English-Oromo Neural Machine Translation using Community Engagement Platformcited by 4
  14. 2023The African Stopwords project: curating stopwords for African languagescited by 3
  15. 2025Tell, Don't Show: Leveraging Language Models' Abstractive Retellings to Model Literary Themescited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Translate With Care: Addressing Gender Bias, Neutrality, and Reasoning in Large Language Model Translations
  2. 2025An LLM-as-a-judge Approach for Scalable Gender-Neutral Translation Evaluation
  3. 2025Probing Gender Bias in Multilingual LLMs: A Case Study of Stereotypes in Persian
  4. 2023Women Are Beautiful, Men Are Leaders: Gender Stereotypes in Machine Translation and Language Modeling
  5. 2025EuroGEST: Investigating gender stereotypes in multilingual language models
  6. 2024GenderAlign: An Alignment Dataset for Mitigating Gender Bias in Large Language Models
  7. 2025Dissecting Bias in LLMs: A Mechanistic Interpretability Perspective
  8. 2026A Factorial Study of Synthetic Data Generation for Low-Resource Machine Translation using Grammar Books