# Physician Detection of Clinical Harm in Machine Translation: Quality Estimation Aids in Reliance and Backtranslation Identifies Critical Errors

arXiv: https://arxiv.org/abs/2310.16924 · HTML: https://mlsanity.com/paper/2310.16924/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 52 references that are in the map.
- First posted on arXiv 25 Oct 2023.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance](https://mlsanity.com/paper/2006.14779/) (2020)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanation](https://mlsanity.com/paper/1802.00682/) (2018)
- [Explainable AI is Dead, Long Live Explainable AI! Hypothesis-driven decision support](https://mlsanity.com/paper/2302.12389/) (2023)
- [TransQuest: Translation Quality Estimation with Cross-lingual Transformers](https://mlsanity.com/paper/2011.01536/) (2020)
- [OpenKiwi: An Open Source Framework for Quality Estimation](https://mlsanity.com/paper/1902.08646/) (2019)
- [Uncalibrated Models Can Improve Human-AI Collaboration](https://mlsanity.com/paper/2202.05983/) (2022)
- [Translation Quality Assessment: A Brief Survey on Manual and Automatic Methods](https://mlsanity.com/paper/2105.03311/) (2021)
- [Backtranslation Feedback Improves User Confidence in MT, Not Quality](https://mlsanity.com/paper/2104.05688/) (2021)

## Built on by (most cited first, 22 of 22)

- [Can Automatic Metrics Assess High-Quality Translations?](https://mlsanity.com/paper/2405.18348/) (2024)
- [SpeechQE: Estimating the Quality of Direct Speech Translation](https://mlsanity.com/paper/2410.21485/) (2024)
- [Translation in the Hands of Many:Centering Lay Users in Machine Translation Interactions](https://mlsanity.com/paper/2502.13780/) (2025)
- [AskQE: Question Answering as Automatic Evaluation for Machine Translation](https://mlsanity.com/paper/2504.11582/) (2025)
- [QE4PE: Word-level Quality Estimation for Human Post-Editing](https://mlsanity.com/paper/2503.03044/) (2025)
- [An Interdisciplinary Approach to Human-Centered Machine Translation](https://mlsanity.com/paper/2506.13468/) (2025)
- [A Survey of AI Reliance](https://mlsanity.com/paper/2408.03948/) (2024)
- [Should I Share this Translation? Evaluating Quality Feedback for User Reliance on Machine Translation](https://mlsanity.com/paper/2505.24683/) (2025)
- [Automatic Input Rewriting Improves Translation with Large Language Models](https://mlsanity.com/paper/2502.16682/) (2025)
- [Recent Advances, Applications, and Open Challenges in Machine Learning for Health: Reflections from Research Roundtables at ML4H 2023 Symposium](https://mlsanity.com/paper/2403.01628/) (2024)
- [A Context-aware Framework for Translation-mediated Conversations](https://mlsanity.com/paper/2412.04205/) (2024)
- [Toward Machine Translation Literacy: How Lay Users Perceive and Rely on Imperfect Translations](https://mlsanity.com/paper/2510.09994/) (2025)
- [Designing Beyond Language: Sociotechnical Barriers in AI Health Technologies for Limited English Proficiency](https://mlsanity.com/paper/2511.07277/) (2025)
- [FairQE: Multi-Agent Framework for Mitigating Gender Bias in Translation Quality Estimation](https://mlsanity.com/paper/2604.21420/) (2026)
- [Looking under the Wrong Lamppost: On the Limitations of Automated Translation Quality Estimation](https://mlsanity.com/paper/2608.03577/) (2026)
- [Social and Ethical Risks Posed by General-Purpose LLMs for Settling Newcomers in Canada](https://mlsanity.com/paper/2407.20240/) (2024)
- [HALF: Harm-Aware LLM Fairness Evaluation Aligned with Deployment](https://mlsanity.com/paper/2510.12217/) (2025)
- [ChartAttack: Testing the Vulnerability of LLMs to Malicious Prompting in Chart Generation](https://mlsanity.com/paper/2601.12983/) (2026)
- [Ouvia: A User-centered Framework for Measuring Usability of Speech Translation in Real-World Communication Scenarios](https://mlsanity.com/paper/2606.06177/) (2026)
- [Measuring User's Mental Models of Speech Translation in Human-AI Collaboration](https://mlsanity.com/paper/2606.24644/) (2026)
- [LLMs in the Real World: Evaluating "AI" in Emergency Contexts](https://mlsanity.com/paper/2607.00019/) (2026)
- [Biomedical Machine Translation for Low-Resource Arabic-Script Languages via Cross-Lingual Transfer and LoRA Adapter Merging](https://mlsanity.com/paper/2607.22300/) (2026)

## Abstract (arXiv)

A major challenge in the practical use of Machine Translation (MT) is that users lack guidance to make informed decisions about when to rely on outputs. Progress in quality estimation research provides techniques to automatically assess MT quality, but these techniques have primarily been evaluated in vitro by comparison against human judgments outside of a specific context of use. This paper evaluates quality estimation feedback in vivo with a human study simulating decision-making in high-stakes medical settings. Using Emergency Department discharge instructions, we study how interventions based on quality estimation versus backtranslation assist physicians in deciding whether to show MT outputs to a patient. We find that quality estimation improves appropriate reliance on MT, but backtranslation helps physicians detect more clinically harmful errors that QE alone often misses.