mlsanityData to 14 Oct 2026

Embarrassingly Easy Document-Level MT Metrics: How to Convert Any Pretrained Metric Into a Document-Level Metric

arXiv 2209.13654 ↗First posted 27 Sep 2022cs.CLcited by 56

Not on the map. Why?

Abstract (arXiv)

We hypothesize that existing sentence-level machine translation (MT) metrics become less effective when the human reference contains ambiguities. To verify this hypothesis, we present a very simple method for extending pretrained metrics to incorporate context at the document level. We apply our method to three popular metrics, BERTScore, Prism, and COMET, and to the reference free metric COMET-QE. We evaluate the extended metrics on the WMT 2021 metrics shared task using the provided MQM annotations. Our results show that the extended metrics outperform their sentence-level counterparts in about 85% of the tested conditions, when excluding results on low-quality human references. Additionally, we show that our document-level extension of COMET-QE dramatically improves its accuracy on discourse phenomena tasks, outperforming a dedicated baseline by up to 6.1%. Our experimental results support our initial hypothesis and show that a simple extension of the metrics permits them to take advantage of context to resolve ambiguities in the reference.

Key facts. Cited by 56 papers in the map (data through 14 Oct 2026). Builds on 22 of its 48 references that are in the map. First posted on arXiv 27 Sep 2022.

Builds on

22 of 48 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  5. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  6. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  7. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 338
  8. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  9. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 163
  10. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 115
  11. 2018Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluationcited by 88
  12. 2017Evaluating Discourse Phenomena in Neural Machine Translationcited by 87
10 more
  1. 2021A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methodscited by 72
  2. 2021BlonDe: An Automatic Evaluation Metric for Document-level Machine Translationcited by 58
  3. 2021Measuring and Increasing Context Usage in Context-Aware Machine Translationcited by 48
  4. 2019When and Why is Document-level Context Useful in Neural Machine Translation?cited by 45
  5. 2020Paraphrase Generation as Zero-Shot Multilingual Translation: Disentangling Semantic Similarity from Lexical and Syntactic Diversitycited by 35
  6. 2019Mining Discourse Markers for Unsupervised Sentence Representation Learningcited by 34
  7. 2020Reassessing Claims of Human Parity and Super-Human Performance in Machine Translation at WMT 2019cited by 20
  8. 2020Exploiting Sentence Order in Document Alignmentcited by 13
  9. 2019Hierarchical Document Encoder for Parallel Corpus Miningcited by 10
  10. 2019Evaluating Pronominal Anaphora in Machine Translation: An Evaluation Measure and a Test Suitecited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 13; 2025: 22; 2026: 12.

56 papers.

Sort:

By area:

  1. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
  2. 2023Large language models effectively leverage document-level context for literary translation, but critical errors persistcited by 80
  3. 2024(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Textscited by 49
  4. 2024A Shocking Amount of the Web is Machine Translated: Insights from Multi-Way Parallelismcited by 35
  5. 2023Escaping the sentence-level paradigm in machine translationcited by 27
  6. 2024DelTA: An Online Document-Level Translation Agent Based on Multi-Level Memorycited by 27
  7. 2024Improving Statistical Significance in Human Evaluation of Automatic Metrics via Soft Pairwise Accuracycited by 25
  8. 2024Fine-Tuned Machine Translation Metrics Struggle in Unseen Domainscited by 21
  9. 2023Identifying Context-Dependent Translations for Evaluation Set Productioncited by 14
  10. 2024Don't Rank, Combine! Combining Machine Translation Hypotheses Using Quality Estimationcited by 13
  11. 2023SLIDE: Reference-free Evaluation for Machine Translation using a Sliding Document Windowcited by 11
  12. 2025Multilingual Contextualization of Large Language Models for Document-Level Machine Translationcited by 11
38 more
  1. 2024Watching the Watchers: Exposing Gender Disparities in Machine Translation Quality Estimationcited by 10
  2. 2025Do LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answeringcited by 10
  3. 2025Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progresscited by 10
  4. 2023Training and Meta-Evaluating Machine Translation Evaluation Metrics at the Paragraph Levelcited by 9
  5. 2024Adapters for Altering LLM Vocabularies: What Languages Benefit the Most?cited by 9
  6. 2025Science Across Languages: Assessing LLM Multilingual Translation of Scientific Paperscited by 8
  7. 2023Discourse Centric Evaluation of Machine Translation with a Densely Annotated Parallel Corpuscited by 7
  8. 2024Quality and Quantity of Machine Translation References for Automatic Metricscited by 7
  9. 2024Recovering document annotations for sentence-level bitextcited by 6
  10. 2025Extending Automatic Machine Translation Evaluation to Book-Length Documentscited by 6
  11. 2023Challenges in Context-Aware Neural Machine Translationcited by 5
  12. 2023On-the-Fly Fusion of Large Language Models and Machine Translationcited by 5
  13. 2025Two Intermediate Translations Are Better Than One: Fine-tuning LLMs for Document-level Translation Refinementcited by 5
  14. 2024M3T: A New Benchmark Dataset for Multi-Modal Document-Level Machine Translationcited by 4
  15. 2025Specification-Aware Machine Translation and Evaluation for Purpose Alignmentcited by 4
  16. 2024MQM-Chat: Multidimensional Quality Metrics for Chat Translationcited by 3
  17. 2024A Context-aware Framework for Translation-mediated Conversationscited by 2
  18. 2025Test-Time Alignment for Large Language Models via Textual Model Predictive Controlcited by 2
  19. 2025Improving LLM-based Document-level Machine Translation with Multi-Knowledge Fusioncited by 2
  20. 2025GRAFT: A Graph-based Flow-aware Agentic Framework for Document-level Machine Translationcited by 2
  21. 2025Align-then-Slide: A complete evaluation framework for Ultra-Long Document-Level Machine Translationcited by 2
  22. 2025Long-context Reference-based MT Quality Estimationcited by 2
  23. 2025Non-Linear Scoring Model for Translation Quality Evaluationcited by 2
  24. 2026Cross-Preference Learning for Sentence-Level and Context-Aware Machine Translationcited by 2
  25. 2025Automatic Evaluation Metrics for Document-level Translation: Overview, Challenges and Trendscited by 1
  26. 2025Same evaluation, more tokens: On the effect of input length for machine translation evaluation using Large Language Modelscited by 1
  27. 2025TransBench: Benchmarking Machine Translation for Industrial-Scale Applicationscited by 1
  28. 2025Quality Estimation Reranking for Document-Level Translationcited by 1
  29. 2025ACADATA: Parallel Dataset of Academic Data for Machine Translationcited by 1
  30. 2025Discourse Graph Guided Document Translation with Large Language Modelscited by 1
  31. 2026Loong: A Human-Like Long Document Translation Agent with Observe-and-Act Adaptive Context Selectioncited by 1
  32. 2026Which Tokens Need Context? A Reference-Based Analysis of Translation Responsibility Using Fertility and Entropycited by 1
  33. 2026STAR : Sentence Translation Alignment Rate for Document-to-Document Machine Translationcited by 1
  34. 2025Interpretable Style Takagi-Sugeno-Kang Fuzzy Clusteringcited by 0
  35. 2025Document-Level Text Generation with Minimum Bayes Risk Decoding using Optimal Transportcited by 0
  36. 2025Structured Document Translation via Format Reinforcement Learningcited by 0
  37. 2026Enhancing Document-Level Machine Translation via Filtered Synthetic Corpora and Two-Stage LLM Adaptationcited by 0
  38. 2026G^2C-MT: Graph-Guided Context Selection for Document-Level Machine Translationcited by 0

50 most cited of 56 All 56 (JSON)