# You Cannot Feed Two Birds with One Score: the Accuracy-Naturalness Tradeoff in Translation

arXiv: https://arxiv.org/abs/2503.24013 · HTML: https://mlsanity.com/paper/2503.24013/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 45 references that are in the map.
- First posted on arXiv 31 Mar 2025.

## Builds on (most cited first, 17 of 17)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [The Perception-Distortion Tradeoff](https://mlsanity.com/paper/1711.06077/) (2017)
- [High-Fidelity Generative Image Compression](https://mlsanity.com/paper/2006.09965/) (2020)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [Rethinking Lossy Compression: The Rate-Distortion-Perception Tradeoff](https://mlsanity.com/paper/1901.07821/) (2019)
- [Input complexity and out-of-distribution detection with likelihood-based generative models](https://mlsanity.com/paper/1909.11480/) (2019)
- [Language GANs Falling Short](https://mlsanity.com/paper/1811.02549/) (2018)
- [MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task](https://mlsanity.com/paper/2410.03983/) (2024)
- [Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translation](https://mlsanity.com/paper/2102.00287/) (2021)
- [Controlled Text Generation as Continuous Optimization with Multiple Constraints](https://mlsanity.com/paper/2108.01850/) (2021)
- [A Set of Recommendations for Assessing Human-Machine Parity in Language Translation](https://mlsanity.com/paper/2004.01694/) (2020)
- [Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judge](https://mlsanity.com/paper/2410.03775/) (2024)
- [A coding theorem for the rate-distortion-perception function](https://mlsanity.com/paper/2104.13662/) (2021)
- [BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Training](https://mlsanity.com/paper/2307.03131/) (2023)
- [What makes an image realistic?](https://mlsanity.com/paper/2403.04493/) (2024)
- [Simpson's Paradox and the Accuracy-Fluency Tradeoff in Translation](https://mlsanity.com/paper/2402.12690/) (2024)

## Built on by (most cited first, 4 of 4)

- [Feeding Two Birds or Favoring One? Adequacy-Fluency Tradeoffs in Evaluation and Meta-Evaluation of Machine Translation](https://mlsanity.com/paper/2509.20287/) (2025)
- [Translationese-index: Using Likelihood Ratios for Graded and Generalizable Measurement of Translationese](https://mlsanity.com/paper/2507.12260/) (2025)
- [LLMs in the Real World: Evaluating "AI" in Emergency Contexts](https://mlsanity.com/paper/2607.00019/) (2026)
- [Mind Which Bird You Favour: Parameterizing Adequacy-Fluency Balance in Meta-Evaluation of Machine Translation](https://mlsanity.com/paper/2609.14795/) (2026)

## Abstract (arXiv)

The goal of translation, be it by human or by machine, is, given some text in a source language, to produce text in a target language that simultaneously 1) preserves the meaning of the source text and 2) achieves natural expression in the target language. However, researchers in the machine translation community usually assess translations using a single score intended to capture semantic accuracy and the naturalness of the output simultaneously. In this paper, we build on recent advances in information theory to mathematically prove and empirically demonstrate that such single-score summaries do not and cannot give the complete picture of a system's true performance. Concretely, we prove that a tradeoff exists between accuracy and naturalness and demonstrate it by evaluating the submissions to the WMT24 shared task. Our findings help explain well-known empirical phenomena, such as the observation that optimizing translation systems for a specific accuracy metric (like BLEU) initially improves the system's naturalness, while ``overfitting'' the system to the metric can significantly degrade its naturalness. Thus, we advocate for a change in how translations are evaluated: rather than comparing systems using a single number, they should be compared on an accuracy-naturalness plane.