# Cyber Risks of Machine Translation Critical Errors : Arabic Mental Health Tweets as a Case Study

arXiv: https://arxiv.org/abs/2405.11668 · HTML: https://mlsanity.com/paper/2405.11668/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 50 references that are in the map.
- First posted on arXiv 19 May 2024.

## Builds on (most cited first, 12 of 12)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Six Challenges for Neural Machine Translation](https://mlsanity.com/paper/1706.03872/) (2017)
- [Suicidal Ideation Detection: A Review of Machine Learning Methods and Applications](https://mlsanity.com/paper/1910.12611/) (2019)
- [Toxicity in Multilingual Machine Translation at Scale](https://mlsanity.com/paper/2210.03070/) (2022)
- [BLEU, METEOR, BERTScore: Evaluation of Metrics Performance in Assessing Critical Translation Errors in Sentiment-oriented Text](https://mlsanity.com/paper/2109.14250/) (2021)
- [Fluency Over Adequacy: A Pilot Study in Measuring User Trust in Imperfect MT](https://mlsanity.com/paper/1802.06041/) (2018)
- [The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metrics](https://mlsanity.com/paper/2305.11806/) (2023)
- [Is it Great or Terrible? Preserving Sentiment in Neural Machine Translation of Arabic Reviews](https://mlsanity.com/paper/2010.13814/) (2020)
- [Challenges in Translation of Emotions in Multilingual User-Generated Content: Twitter as a Case Study](https://mlsanity.com/paper/2106.10719/) (2021)
- [Sentiment-Aware Measure (SAM) for Evaluating Sentiment Transfer by Machine Translation Systems](https://mlsanity.com/paper/2109.14895/) (2021)

## Abstract (arXiv)

With the advent of Neural Machine Translation (NMT) systems, the MT output has reached unprecedented accuracy levels which resulted in the ubiquity of MT tools on almost all online platforms with multilingual content. However, NMT systems, like other state-of-the-art AI generative systems, are prone to errors that are deemed machine hallucinations. The problem with NMT hallucinations is that they are remarkably textitfluent hallucinations. Since they are trained to produce grammatically correct utterances, NMT systems are capable of producing mistranslations that are too fluent to be recognised by both users of the MT tool, as well as by automatic quality metrics that are used to gauge their performance. In this paper, we introduce an authentic dataset of machine translation critical errors to point to the ethical and safety issues involved in the common use of MT. The dataset comprises mistranslations of Arabic mental health postings manually annotated with critical error types. We also show how the commonly used quality metrics do not penalise critical errors and highlight this as a critical issue that merits further attention from researchers.