mlsanityData to 7 Oct 2026

Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing

arXiv 2004.14564 ↗First posted 30 Apr 2020cs.CLcited by 114

Not on the map. Why?

Abstract (arXiv)

We frame the task of machine translation evaluation as one of scoring machine translation output with a sequence-to-sequence paraphraser, conditioned on a human reference. We propose training the paraphraser as a multilingual NMT system, treating paraphrasing as a zero-shot translation task (e.g., Czech to Czech). This results in the paraphraser's output mode being centered around a copy of the input sequence, which represents the best case scenario where the MT system output matches a human reference. Our method is simple and intuitive, and does not require human judgements for training. Our single model (trained in 39 languages) outperforms or statistically ties with all prior metrics on the WMT 2019 segment-level shared metrics task in all languages (excluding Gujarati where the model had no training data). We also explore using our model for the task of quality estimation as a metric--conditioning on the source instead of the reference--and find that it significantly outperforms every submission to the WMT 2019 shared task on quality estimation in every language pair.

Key facts. Cited by 114 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 36; 2023: 32; 2024: 24; 2025: 7; 2026: 1.

114 papers.

Sort:

By area:

  1. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  2. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  3. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  4. 2023GPTScore: Evaluate as You Desirecited by 357
  5. 2022Out-of-Distribution Detection and Selective Generation for Conditional Language Modelscited by 193
  6. 2022ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoningcited by 175
  7. 2022In-context Examples Selection for Machine Translationcited by 172
  8. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  9. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 161
  10. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  11. 2021Learning Compact Metrics for MTcited by 93
  12. 2024MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Taskcited by 92
38 more
  1. 2023Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Modelscited by 87
  2. 2023Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Studycited by 85
  3. 2023ReCEval: Evaluating Reasoning Chains via Correctness and Informativenesscited by 76
  4. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 72
  5. 2022Embarrassingly Easy Document-Level MT Metrics: How to Convert Any Pretrained Metric Into a Document-Level Metriccited by 56
  6. 2023Calibrating LLM-Based Evaluatorcited by 53
  7. 2022On the Limitations of Reference-Free Evaluations of Generated Textcited by 52
  8. 2022BERTScore is Unfair: On Social Bias in Language Model-Based Metrics for Text Generationcited by 48
  9. 2022On the Blind Spots of Model-Based Evaluation Metrics for Text Generationcited by 48
  10. 2022Exploring Document-Level Literary Machine Translation with Parallel Paragraphs from World Literaturecited by 45
  11. 2021Automatic Text Evaluation through the Lens of Wasserstein Barycenterscited by 38
  12. 2024Leveraging Large Language Models for NLG Evaluation: Advances and Challengescited by 37
  13. 2022DEMETR: Diagnosing Evaluation Metrics for Translationcited by 36
  14. 2024X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at Scalecited by 36
  15. 2021Uncertainty-Aware Machine Translation Evaluationcited by 35
  16. 2022DiscoScore: Evaluating Text Generation with BERT and Discourse Coherencecited by 35
  17. 2024A Shocking Amount of the Web is Machine Translated: Insights from Multi-Way Parallelismcited by 33
  18. 2024HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decompositioncited by 33
  19. 2021Bidimensional Leaderboards: Generate and Evaluate Language Hand in Handcited by 32
  20. 2022UniTE: Unified Translation Evaluationcited by 28
  21. 2021Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translationcited by 27
  22. 2022IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation metrics for Indian Languagescited by 27
  23. 2024Improving Statistical Significance in Human Evaluation of Automatic Metrics via Soft Pairwise Accuracycited by 25
  24. 2023Gender Biases in Automatic Evaluation Metrics for Image Captioningcited by 24
  25. 2023BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Trainingcited by 24
  26. 2023The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metricscited by 23
  27. 2023INSTRUCTSCORE: Explainable Text Generation Evaluation with Finegrained Feedbackcited by 21
  28. 2024Fine-Tuned Machine Translation Metrics Struggle in Unseen Domainscited by 21
  29. 2024Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!cited by 21
  30. 2024IdeaBench: Benchmarking Large Language Models for Research Idea Generationcited by 19
  31. 2022MENLI: Robust Evaluation Metrics from Natural Language Inferencecited by 18
  32. 2022Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error Synthesiscited by 18
  33. 2022T5Score: Discriminative Fine-tuning of Generative Evaluation Metricscited by 18
  34. 2022Toward Human-Like Evaluation for Natural Language Generation with Error Analysiscited by 18
  35. 2022SMART: Sentences as Basic Units for Text Evaluationcited by 17
  36. 2022RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Questioncited by 17
  37. 2022Dubbing in Practice: A Large Scale Study of Human Localization With Insights for Automatic Dubbingcited by 17
  38. 2021Everything Is All It Takes: A Multipronged Strategy for Zero-Shot Cross-Lingual Information Extractioncited by 16

50 most cited of 114 All 114 (JSON)