mlsanityData to 7 Oct 2026

Fluency Over Adequacy: A Pilot Study in Measuring User Trust in Imperfect MT

arXiv 1802.06041 ↗First posted 16 Feb 2018cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

Although measuring intrinsic quality has been a key factor in the advancement of Machine Translation (MT), successfully deploying MT requires considering not just intrinsic quality but also the user experience, including aspects such as trust. This work introduces a method of studying how users modulate their trust in an MT system after seeing errorful (disfluent or inadequate) output amidst good (fluent and adequate) output. We conduct a survey to determine how users respond to good translations compared to translations that are either adequate but not fluent, or fluent but not adequate. In this pilot study, users responded strongly to disfluent translations, but were, surprisingly, much less concerned with adequacy.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 1; 2023: 6; 2024: 6; 2025: 3; 2026: 4.

21 papers.

Sort:

By area:

  1. 2021Gender Bias in Machine Translationcited by 116
  2. 2024Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translationcited by 62
  3. 2023Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspectioncited by 44
  4. 2024Translating Step-by-Step: Decomposing the Translation Process for Improved Translation Quality of Long-Form Textscited by 31
  5. 2023Why LLMs Hallucinate, and How to Get (Evidential) Closure: Perceptual, Intensional, and Extensional Learning for Faithful Natural Language Generationcited by 13
  6. 2024Simpson's Paradox and the Accuracy-Fluency Tradeoff in Translationcited by 8
  7. 2024Modeling User Preferences with Automatic Metrics: Creating a High-Quality Preference Dataset for Machine Translationcited by 8
  8. 2023LLMMaps -- A Visual Metaphor for Stratified Evaluation of Large Language Modelscited by 6
  9. 2024ToBlend: Token-Level Blending With an Ensemble of LLMs to Attack AI-Generated Text Detectioncited by 6
  10. 2025An Interdisciplinary Approach to Human-Centered Machine Translationcited by 6
  11. 2023First Tragedy, then Parse: History Repeats Itself in the New Era of Large Language Modelscited by 4
  12. 2025Feeding Two Birds or Favoring One? Adequacy-Fluency Tradeoffs in Evaluation and Meta-Evaluation of Machine Translationcited by 3
9 more
  1. 2023Unsupervised Approach to Evaluate Sentence-Level Fluency: Do We Really Need Reference?cited by 2
  2. 2022Beyond General Purpose Machine Translation: The Need for Context-specific Empirical Research to Design for Appropriate User Trustcited by 1
  3. 2023Participatory Research as a Path to Community-Informed, Gender-Fair Machine Translationcited by 1
  4. 2025Toward Machine Translation Literacy: How Lay Users Perceive and Rely on Imperfect Translationscited by 1
  5. 2024Cyber Risks of Machine Translation Critical Errors : Arabic Mental Health Tweets as a Case Studycited by 0
  6. 2026Blinded Radiologist and LLM-Based Evaluation of LLM-Generated Japanese Translations of Chest CT Reports: Comparative Studycited by 0
  7. 2026Measuring User's Mental Models of Speech Translation in Human-AI Collaborationcited by 0
  8. 2026When Readability and Source Retention Diverge: An Evaluability Gap in AI Translationcited by 0
  9. 2026Mind Which Bird You Favour: Parameterizing Adequacy-Fluency Balance in Meta-Evaluation of Machine Translationcited by 0