mlsanityData to 7 Oct 2026

Poor Man's Quality Estimation: Predicting Reference-Based MT Metrics Without the Reference

arXiv 2301.09008 ↗First posted 21 Jan 2023cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Machine translation quality estimation (QE) predicts human judgements of a translation hypothesis without seeing the reference. State-of-the-art QE systems based on pretrained language models have been achieving remarkable correlations with human judgements yet they are computationally heavy and require human annotations, which are slow and expensive to create. To address these limitations, we define the problem of metric estimation (ME) where one predicts the automated metric scores also without the reference. We show that even without access to the reference, our model can estimate automated metrics (ρ=60% for BLEU, ρ=51% for other metrics) at the sentence-level. Because automated metrics correlate with human judgements, we can leverage the ME task for pre-training a QE model. For the QE task, we find that pre-training on TER is better (ρ=23%) than training for scratch (ρ=20%).

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 32 of its 79 references that are in the map. First posted on arXiv 21 Jan 2023.

Builds on

32 of 79 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  7. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  8. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  9. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  10. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  11. 2017Convolutional Sequence to Sequence Learningcited by 550
  12. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
20 more
  1. 2020Unsupervised Quality Estimation for Neural Machine Translationcited by 210
  2. 2018Scaling Neural Machine Translationcited by 204
  3. 2019Facebook FAIR's WMT19 News Translation Task Submissioncited by 179
  4. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  5. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 161
  6. 2021Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answeringcited by 131
  7. 2017RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systemscited by 88
  8. 2020GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systemscited by 76
  9. 2020On the Inference Calibration of Neural Machine Translationcited by 61
  10. 2020Unbabel's Participation in the WMT20 Metrics Shared Taskcited by 56
  11. 2019OpenKiwi: An Open Source Framework for Quality Estimationcited by 49
  12. 2022Quality-Aware Decoding for Neural Machine Translationcited by 49
  13. 2021Uncertainty-Aware Machine Translation Evaluationcited by 35
  14. 2022UniTE: Unified Translation Evaluationcited by 28
  15. 2020Self-Paced Learning for Neural Machine Translationcited by 27
  16. 2021Backtranslation Feedback Improves User Confidence in MT, Not Qualitycited by 10
  17. 2022PreQuEL: Quality Estimation of Machine Translation Outputs in Advancecited by 9
  18. 2021DirectQE: Direct Pretraining for Machine Translation Quality Estimationcited by 7
  19. 2020Data Troubles in Sentence Level Confidence Estimation for Machine Translationcited by 3
  20. 2022Fusing Sentence Embeddings Into LSTM-based Autoregressive Language Modelscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 2; 2025: 3; 2026: 2.

11 papers.

  1. 2023Interactive Natural Language Processingcited by 31
  2. 2023A Diachronic Perspective on User Trust in AI under Uncertaintycited by 23
  3. 2025How to Select Datapoints for Efficient Human Evaluation of NLG Models?cited by 23
  4. 2024Fine-Tuned Machine Translation Metrics Struggle in Unseen Domainscited by 21
  5. 2023Towards Explainable Evaluation Metrics for Machine Translationcited by 15
  6. 2024Pitfalls and Outlooks in Using COMETcited by 10
  7. 2025Reference-free Evaluation Metrics for Text Generation: A Surveycited by 9
  8. 2023Tailoring Domain Adaptation for Machine Translation Quality Estimationcited by 4
  9. 2025Are Generative Models Underconfident? Better Quality Estimation with Boosted Model Probabilitycited by 4
  10. 2026Why We Need Speech to Evaluate Speech Translationcited by 1
  11. 2026LocQE: Principled Domain Adaptation for Localisation Quality Estimation by Leveraging Post-Editscited by 0