mlsanityData to 14 Oct 2026

MiTTenS: A Dataset for Evaluating Gender Mistranslation

arXiv 2401.06935 ↗First posted 13 Jan 2024cs.CL cs.CYcited by 5

Not on the map. Why?

Abstract (arXiv)

Translation systems, including foundation models capable of translation, can produce errors that result in gender mistranslation, and such errors can be especially harmful. To measure the extent of such potential harms when translating into and out of English, we introduce a dataset, MiTTenS, covering 26 languages from a variety of language families and scripts, including several traditionally under-represented in digital resources. The dataset is constructed with handcrafted passages that target known failure patterns, longer synthetically generated passages, and natural passages sourced from multiple domains. We demonstrate the usefulness of the dataset by evaluating both neural machine translation systems and foundation models, and show that all systems exhibit gender mistranslation and potential harm, even in high resource languages.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 28 of its 38 references that are in the map. First posted on arXiv 13 Jan 2024.

Builds on

28 of 38 references.

References · 28

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  3. 2021Ethical and social risks of harm from Language Modelscited by 966
  4. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  5. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  6. 2018Gender Bias in Coreference Resolutioncited by 442
  7. 2019Mitigating Gender Bias in Natural Language Processing: Literature Reviewcited by 287
  8. 2019Evaluating Gender Bias in Machine Translationcited by 249
  9. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 195
  10. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 157
  11. 2020Intrinsic Bias Metrics Do Not Correlate with Application Biascited by 148
  12. 2022Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reductioncited by 138
16 more
  1. 2021Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologiescited by 128
  2. 2021Gender Bias in Machine Translationcited by 117
  3. 2019Getting Gender Right in Neural Machine Translationcited by 98
  4. 2021A Survey on Gender Bias in Natural Language Processingcited by 89
  5. 2019Toward Gender-Inclusive Coreference Resolutioncited by 84
  6. 2019On Measuring Gender Bias in Translation of Gender-neutral Pronounscited by 60
  7. 2020Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpuscited by 57
  8. 2023MISGENDERED: Limits of Large Language Models in Understanding Pronounscited by 43
  9. 2021Gender Bias Amplification During Speed-Quality Optimization in Neural Machine Translationcited by 28
  10. 2023An In-depth Look at Gemini's Language Abilitiescited by 28
  11. 2022Toxicity in Multilingual Machine Translation at Scalecited by 26
  12. 2023Multilingual Holistic Bias: Extending Descriptors and Patterns to Unveil Demographic Biases in Languages at Scalecited by 24
  13. 2023What about em? How Commercial Machine Translation Fails to Handle (Neo-)Pronounscited by 19
  14. 2021SynthBio: A Case Study in Human-AI Collaborative Curation of Text Datasetscited by 18
  15. 2021The Arabic Parallel Gender Corpus 2.0: Extensions and Analysescited by 12
  16. 2023Gender Inflected or Bias Inflicted: On Using Grammatical Gender Cues for Bias Evaluation in Machine Translationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 2.

5 papers.

  1. 2024Multilingual Text-to-Image Generation Magnifies Gender Stereotypes and Prompt Engineering May Not Help Youcited by 15
  2. 2025A Multilingual, Culture-First Approach to Addressing Misgendering in LLM Applicationscited by 6
  3. 2025Uncertainty Quantification for Evaluating Machine Translation Biascited by 2
  4. 2026Cultural Fidelity in English-to-Hindi Translation: A Preservation-Fluency Frontier for Gender Recoverabilitycited by 0
  5. 2026Explaining GAND: A Resource on Gender-Ambiguous Natural Data & Contrastive Attributioncited by 0