# Fluency Over Adequacy: A Pilot Study in Measuring User Trust in Imperfect MT

arXiv: https://arxiv.org/abs/1802.06041 · HTML: https://mlsanity.com/paper/1802.06041/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Feb 2018.

## Built on by (most cited first, 21 of 21)

- [Gender Bias in Machine Translation](https://mlsanity.com/paper/2104.06001/) (2021)
- [Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation](https://mlsanity.com/paper/2406.11580/) (2024)
- [Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspection](https://mlsanity.com/paper/2301.07779/) (2023)
- [Translating Step-by-Step: Decomposing the Translation Process for Improved Translation Quality of Long-Form Texts](https://mlsanity.com/paper/2409.06790/) (2024)
- [Why LLMs Hallucinate, and How to Get (Evidential) Closure: Perceptual, Intensional, and Extensional Learning for Faithful Natural Language Generation](https://mlsanity.com/paper/2310.15355/) (2023)
- [Simpson's Paradox and the Accuracy-Fluency Tradeoff in Translation](https://mlsanity.com/paper/2402.12690/) (2024)
- [Modeling User Preferences with Automatic Metrics: Creating a High-Quality Preference Dataset for Machine Translation](https://mlsanity.com/paper/2410.07779/) (2024)
- [LLMMaps -- A Visual Metaphor for Stratified Evaluation of Large Language Models](https://mlsanity.com/paper/2304.00457/) (2023)
- [ToBlend: Token-Level Blending With an Ensemble of LLMs to Attack AI-Generated Text Detection](https://mlsanity.com/paper/2402.11167/) (2024)
- [An Interdisciplinary Approach to Human-Centered Machine Translation](https://mlsanity.com/paper/2506.13468/) (2025)
- [First Tragedy, then Parse: History Repeats Itself in the New Era of Large Language Models](https://mlsanity.com/paper/2311.05020/) (2023)
- [Feeding Two Birds or Favoring One? Adequacy-Fluency Tradeoffs in Evaluation and Meta-Evaluation of Machine Translation](https://mlsanity.com/paper/2509.20287/) (2025)
- [Unsupervised Approach to Evaluate Sentence-Level Fluency: Do We Really Need Reference?](https://mlsanity.com/paper/2312.01500/) (2023)
- [Beyond General Purpose Machine Translation: The Need for Context-specific Empirical Research to Design for Appropriate User Trust](https://mlsanity.com/paper/2205.06920/) (2022)
- [Participatory Research as a Path to Community-Informed, Gender-Fair Machine Translation](https://mlsanity.com/paper/2306.08906/) (2023)
- [Toward Machine Translation Literacy: How Lay Users Perceive and Rely on Imperfect Translations](https://mlsanity.com/paper/2510.09994/) (2025)
- [Cyber Risks of Machine Translation Critical Errors : Arabic Mental Health Tweets as a Case Study](https://mlsanity.com/paper/2405.11668/) (2024)
- [Blinded Radiologist and LLM-Based Evaluation of LLM-Generated Japanese Translations of Chest CT Reports: Comparative Study](https://mlsanity.com/paper/2604.02207/) (2026)
- [Measuring User's Mental Models of Speech Translation in Human-AI Collaboration](https://mlsanity.com/paper/2606.24644/) (2026)
- [When Readability and Source Retention Diverge: An Evaluability Gap in AI Translation](https://mlsanity.com/paper/2608.19083/) (2026)
- [Mind Which Bird You Favour: Parameterizing Adequacy-Fluency Balance in Meta-Evaluation of Machine Translation](https://mlsanity.com/paper/2609.14795/) (2026)

## Abstract (arXiv)

Although measuring intrinsic quality has been a key factor in the advancement of Machine Translation (MT), successfully deploying MT requires considering not just intrinsic quality but also the user experience, including aspects such as trust. This work introduces a method of studying how users modulate their trust in an MT system after seeing errorful (disfluent or inadequate) output amidst good (fluent and adequate) output. We conduct a survey to determine how users respond to good translations compared to translations that are either adequate but not fluent, or fluent but not adequate. In this pilot study, users responded strongly to disfluent translations, but were, surprisingly, much less concerned with adequacy.