mlsanityData to 7 Oct 2026

Assessing Reference-Free Peer Evaluation for Machine Translation

arXiv 2104.05146 ↗First posted 12 Apr 2021cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

Reference-free evaluation has the potential to make machine translation evaluation substantially more scalable, allowing us to pivot easily to new languages or domains. It has been recently shown that the probabilities given by a large, multilingual model can achieve state of the art results when used as a reference-free metric. We experiment with various modifications to this model and demonstrate that by scaling it up we can match the performance of BLEU. We analyze various potential weaknesses of the approach and find that it is surprisingly robust and likely to offer reasonable performance across a broad spectrum of domains and different system qualities.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 23 of its 36 references that are in the map. First posted on arXiv 12 Apr 2021.

Builds on

23 of 36 references.

References · 23

  1. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  2. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  3. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  4. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  5. 2020Unsupervised Quality Estimation for Neural Machine Translationcited by 210
  6. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 186
  7. 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 155
  8. 2020With Little Power Comes Great Responsibilitycited by 138
  9. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 114
  10. 2020BLEU might be Guilty but References are not Innocentcited by 109
  11. 2019Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modelingcited by 86
  12. 2018Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddingscited by 83
11 more
  1. 2019Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmaxcited by 58
  2. 2020Learning an Unreferenced Metric for Online Dialogue Evaluationcited by 52
  3. 2020Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translationcited by 51
  4. 2020On the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluationcited by 36
  5. 2020Reassessing Claims of Human Parity and Super-Human Performance in Machine Translation at WMT 2019cited by 20
  6. 2016The Neural Noisy Channelcited by 19
  7. 2016There's No Comparison: Reference-less Evaluation Metrics in Grammatical Error Correctioncited by 18
  8. 2020Parallel Corpus Filtering via Pre-trained Language Modelscited by 15
  9. 2020Human-Paraphrased References Improve Neural Machine Translationcited by 15
  10. 2020BLEU Neighbors: A Reference-less Approach to Automatic Evaluationcited by 3
  11. 2020Data Troubles in Sentence Level Confidence Estimation for Machine Translationcited by 3

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 3; 2023: 0; 2024: 1; 2025: 3.

8 papers.

  1. 2022In-context Examples Selection for Machine Translationcited by 172
  2. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 161
  3. 2022On the Limitations of Reference-Free Evaluations of Generated Textcited by 52
  4. 2024From One to Many: Expanding the Scope of Toxicity Mitigation in Language Modelscited by 18
  5. 2025Beyond Factual Accuracy: Evaluating Coverage of Diverse Factual Information in Long-form Text Generationcited by 15
  6. 2025Reference-free Evaluation Metrics for Text Generation: A Surveycited by 9
  7. 2022NMTScore: A Multilingual Analysis of Translation-based Text Similarity Measurescited by 6
  8. 2025Feeding Two Birds or Favoring One? Adequacy-Fluency Tradeoffs in Evaluation and Meta-Evaluation of Machine Translationcited by 3