mlsanityData to 14 Oct 2026

Training and Meta-Evaluating Machine Translation Evaluation Metrics at the Paragraph Level

arXiv 2308.13506 ↗First posted 25 Aug 2023cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

As research on machine translation moves to translating text beyond the sentence level, it remains unclear how effective automatic evaluation metrics are at scoring longer translations. In this work, we first propose a method for creating paragraph-level data for training and meta-evaluating metrics from existing sentence-level data. Then, we use these new datasets to benchmark existing sentence-level metrics as well as train learned metrics at the paragraph level. Interestingly, our experimental results demonstrate that using sentence-level metrics to score entire paragraphs is equally as effective as using a metric designed to work at the paragraph level. We speculate this result can be attributed to properties of the task of reference-based evaluation as well as limitations of our datasets with respect to capturing all types of phenomena that occur in paragraph-level translations.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 25 of its 47 references that are in the map. First posted on arXiv 25 Aug 2023.

Builds on

25 of 47 references.

References · 25

  1. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  2. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  3. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  4. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  5. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  6. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  7. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
  8. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 338
  9. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  10. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 163
  11. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 115
  12. 2017Neural Machine Translation with Extended Contextcited by 95
13 more
  1. 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
  2. 2017Evaluating Discourse Phenomena in Neural Machine Translationcited by 87
  3. 2018Document-Level Neural Machine Translation with Hierarchical Attention Networkscited by 85
  4. 2023Large language models effectively leverage document-level context for literary translation, but critical errors persistcited by 80
  5. 2023Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibrationcited by 74
  6. 2021BlonDe: An Automatic Evaluation Metric for Document-level Machine Translationcited by 58
  7. 2022Embarrassingly Easy Document-Level MT Metrics: How to Convert Any Pretrained Metric Into a Document-Level Metriccited by 56
  8. 2022Exploring Document-Level Literary Machine Translation with Parallel Paragraphs from World Literaturecited by 45
  9. 2017Learning to Remember Translation History with a Continuous Cachecited by 39
  10. 2020Rethinking Document-level Neural Machine Translationcited by 32
  11. 2022UniTE: Unified Translation Evaluationcited by 28
  12. 2023Escaping the sentence-level paradigm in machine translationcited by 27
  13. 2023Document Flattening: Beyond Concatenating Context for Document-Level Neural Machine Translationcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 4; 2025: 2; 2026: 1.

9 papers.

  1. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 265
  2. 2024Improving Statistical Significance in Human Evaluation of Automatic Metrics via Soft Pairwise Accuracycited by 25
  3. 2024Efficient Minimum Bayes Risk Decoding using Low-Rank Matrix Completion Algorithmscited by 13
  4. 2023SLIDE: Reference-free Evaluation for Machine Translation using a Sliding Document Windowcited by 11
  5. 2025Do LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answeringcited by 10
  6. 2024Adapters for Altering LLM Vocabularies: What Languages Benefit the Most?cited by 9
  7. 2024Towards Chapter-to-Chapter Context-Aware Literary Translation via Large Language Modelscited by 2
  8. 2025ACADATA: Parallel Dataset of Academic Data for Machine Translationcited by 1
  9. 2026TermJudge: A Document-Level Metric Judging, Not Counting, Terminology in Machine Translation Evaluationcited by 0