# Feeding Two Birds or Favoring One? Adequacy-Fluency Tradeoffs in Evaluation and Meta-Evaluation of Machine Translation

arXiv: https://arxiv.org/abs/2509.20287 · HTML: https://mlsanity.com/paper/2509.20287/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 46 references that are in the map.
- First posted on arXiv 24 Sep 2025.

## Builds on (most cited first, 17 of 17)

- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection](https://mlsanity.com/paper/2310.10482/) (2023)
- [To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation](https://mlsanity.com/paper/2107.10821/) (2021)
- [MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task](https://mlsanity.com/paper/2410.03983/) (2024)
- [RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems](https://mlsanity.com/paper/1701.03079/) (2017)
- [Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Task](https://mlsanity.com/paper/2309.11925/) (2023)
- [Beyond BLEU: Training Neural Machine Translation with Semantic Similarity](https://mlsanity.com/paper/1909.06694/) (2019)
- [Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration](https://mlsanity.com/paper/2305.14324/) (2023)
- [Improving Statistical Significance in Human Evaluation of Automatic Metrics via Soft Pairwise Accuracy](https://mlsanity.com/paper/2409.09598/) (2024)
- [Fluency Over Adequacy: A Pilot Study in Measuring User Trust in Imperfect MT](https://mlsanity.com/paper/1802.06041/) (2018)
- [Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!](https://mlsanity.com/paper/2408.13831/) (2024)
- [Machine Translation Meta Evaluation through Translation Accuracy Challenge Sets](https://mlsanity.com/paper/2401.16313/) (2024)
- [Reference-free Evaluation Metrics for Text Generation: A Survey](https://mlsanity.com/paper/2501.12011/) (2025)
- [Assessing Reference-Free Peer Evaluation for Machine Translation](https://mlsanity.com/paper/2104.05146/) (2021)
- [You Cannot Feed Two Birds with One Score: the Accuracy-Naturalness Tradeoff in Translation](https://mlsanity.com/paper/2503.24013/) (2025)

## Built on by (most cited first, 4 of 4)

- [LLMs in the Real World: Evaluating "AI" in Emergency Contexts](https://mlsanity.com/paper/2607.00019/) (2026)
- [Source-Free MT Evaluation Is Not MT Evaluation](https://mlsanity.com/paper/2608.20925/) (2026)
- [Mind Which Bird You Favour: Parameterizing Adequacy-Fluency Balance in Meta-Evaluation of Machine Translation](https://mlsanity.com/paper/2609.14795/) (2026)
- [DUALVIEW: Leveraging Broad and Output-Specific Evidence for LLM-based Multi-Output Text Regression](https://mlsanity.com/paper/2610.19206/) (2026)

## Abstract (arXiv)

We investigate the tradeoff between adequacy and fluency in machine translation. We show the severity of this tradeoff at the evaluation level and analyze where popular metrics fall within it. Essentially, current metrics generally lean toward adequacy, meaning that their scores correlate more strongly with the adequacy of translations than with fluency. More importantly, we find that this tradeoff also persists at the meta-evaluation level, and that the standard WMT meta-evaluation favors adequacy-oriented metrics over fluency-oriented ones. We show that this bias is partially attributed to the composition of the systems included in the meta-evaluation datasets. To control this bias, we propose a method that synthesizes translation systems in meta-evaluation. Our findings highlight the importance of understanding this tradeoff in meta-evaluation and its impact on metric rankings.