mlsanityData to 7 Oct 2026

Toxicity in Multilingual Machine Translation at Scale

arXiv 2210.03070 ↗First posted 6 Oct 2022cs.CLcited by 26

Not on the map. Why?

Abstract (arXiv)

Machine Translation systems can produce different types of errors, some of which are characterized as critical or catastrophic due to the specific negative impact that they can have on users. In this paper we focus on one type of critical error: added toxicity. We evaluate and analyze added toxicity when translating a large evaluation dataset (HOLISTICBIAS, over 472k sentences, covering 13 demographic axes) from English into 164 languages. An automatic toxicity evaluation shows that added toxicity across languages varies from 0% to 5%. The output languages with the most added toxicity tend to be low-resource ones, and the demographic axes with the most added toxicity include sexual orientation, gender and sex, and ability. We also perform human evaluation on a subset of 8 translation directions, confirming the prevalence of true added toxicity. We use a measurement of the amount of source contribution to the translation, where a low source contribution implies hallucination, to interpret what causes toxicity. Making use of the input attributions allows us to explain toxicity, because the source contributions significantly correlate with toxicity for 84% of languages studied. Given our findings, our recommendations to reduce added toxicity are to curate training data to avoid mistranslations, mitigate hallucination and check unstable translations.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 26 of its 50 references that are in the map. First posted on arXiv 6 Oct 2022.

Builds on

26 of 50 references.

References · 26

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  4. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  5. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  6. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  7. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  8. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  9. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  10. 2019On Measuring Social Biases in Sentence Encoderscited by 397
  11. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  12. 2019Measuring Bias in Contextualized Word Representationscited by 275
14 more
  1. 2020Measuring and Reducing Gendered Correlations in Pre-trained Modelscited by 222
  2. 2019Gender Bias in Contextualized Word Embeddingscited by 203
  3. 2019Assessing Social and Intersectional Biases in Contextualized Word Representationscited by 134
  4. 2021Detoxifying Language Models Risks Marginalizing Minority Voicescited by 106
  5. 2022Measuring the Mixing of Contextual Information in the Transformercited by 63
  6. 2021Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarkingcited by 53
  7. 2022Perturbation Augmentation for Fairer NLPcited by 51
  8. 2022Towards Opening the Black Box of Neural Machine Translation: Source and Target Interpretations of the Transformercited by 45
  9. 2022AugLy: Data Augmentations for Robustnesscited by 34
  10. 2020Accurate Word Alignment Induction from Neural Machine Translationcited by 32
  11. 2019Adding Interpretable Attention to Neural Translation Models Improves Word Alignmentcited by 30
  12. 2021Gender Bias Amplification During Speed-Quality Optimization in Neural Machine Translationcited by 28
  13. 2019Saliency-driven Word Alignment Interpretation for Neural Machine Translationcited by 27
  14. 2022Consistent Human Evaluation of Machine Translation across Language Pairscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 4; 2024: 13; 2025: 6; 2026: 1.

26 papers.

Sort:

By area:

  1. 2022Prompting PaLM for Translation: Assessing Strategies and Performancecited by 144
  2. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 123
  3. 2022Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Surveycited by 80
  4. 2023Large language models effectively leverage document-level context for literary translation, but critical errors persistcited by 80
  5. 2025PolyGuard: A Multilingual Safety Moderation Tool for 17 Languagescited by 57
  6. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  7. 2025The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating Itcited by 42
  8. 2023Multilingual Holistic Bias: Extending Descriptors and Patterns to Unveil Demographic Biases in Languages at Scalecited by 23
  9. 2024From One to Many: Expanding the Scope of Toxicity Mitigation in Language Modelscited by 18
  10. 2024PolygloToxicityPrompts: Multilingual Evaluation of Neural Toxic Degeneration in Large Language Modelscited by 16
  11. 2024What the Harm? Quantifying the Tangible Impact of Gender Bias in Machine Translation with a Human-centered Studycited by 15
  12. 2024MuTox: Universal MUltilingual Audio-based TOXicity Dataset and Zero-shot Detectorcited by 14
14 more
  1. 2023Added Toxicity Mitigation at Inference Time for Multimodal and Massively Multilingual Translationcited by 5
  2. 2024MiTTenS: A Dataset for Evaluating Gender Mistranslationcited by 5
  3. 2025ToxicTone: A Mandarin Audio Dataset Annotated for Toxicity and Toxic Utterance Tonalitycited by 5
  4. 2024LCFO: Long Context and Long Form Output Dataset and Benchmarkingcited by 4
  5. 2025Translate, then Detect: Leveraging Machine Translation for Cross-Lingual Toxicity Classificationcited by 4
  6. 2023ReSeTOX: Re-learning attention weights for toxicity mitigation in machine translationcited by 3
  7. 2024On the Role of Speech Data in Reducing Toxicity Detection Biascited by 2
  8. 2025UNITYAI-GUARD: Pioneering Toxicity Detection Across Low-Resource Indian Languagescited by 2
  9. 2024Analyzing Context Contributions in LLM-based Machine Translationcited by 1
  10. 2025Toxicity-Aware Few-Shot Prompting for Low-Resource Singlish Translationcited by 1
  11. 2024Cyber Risks of Machine Translation Critical Errors : Arabic Mental Health Tweets as a Case Studycited by 0
  12. 2024Towards Massive Multilingual Holistic Biascited by 0
  13. 2024MT-LENS: An all-in-one Toolkit for Better Machine Translation Evaluationcited by 0
  14. 2026A Survey of Toxicity Detection and Mitigation Strategies for Multilingual Language Modelscited by 0