mlsanityData to 7 Oct 2026

BlonDe: An Automatic Evaluation Metric for Document-level Machine Translation

arXiv 2103.11878 ↗First posted 22 Mar 2021cs.CL cs.AIcited by 58

Not on the map. Why?

Abstract (arXiv)

Standard automatic metrics, e.g. BLEU, are not reliable for document-level MT evaluation. They can neither distinguish document-level improvements in translation quality from sentence-level ones, nor identify the discourse phenomena that cause context-agnostic translations. This paper introduces a novel automatic metric BlonDe to widen the scope of automatic MT evaluation from sentence to document level. BlonDe takes discourse coherence into consideration by categorizing discourse-related spans and calculating the similarity-based F1 measure of categorized spans. We conduct extensive comparisons on a newly constructed dataset BWB. The experimental results show that BlonDe possesses better selectivity and interpretability at the document-level, and is more sensitive to document-level nuances. In a large-scale human study, BlonDe also achieves significantly higher Pearson's r correlation with human judgments compared to previous metrics.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Builds on 17 of its 67 references that are in the map. First posted on arXiv 22 Mar 2021.

Builds on

17 of 67 references.

References · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  3. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  4. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  5. 2017Six Challenges for Neural Machine Translationcited by 334
  6. 2015Skip-Thought Vectorscited by 321
  7. 2018Achieving Human Parity on Automatic Chinese to English News Translationcited by 119
  8. 2018Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluationcited by 88
  9. 2017Evaluating Discourse Phenomena in Neural Machine Translationcited by 87
  10. 2018Improving the Transformer Translation Model with Document-Level Contextcited by 74
  11. 2017Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generationcited by 61
  12. 2017A Challenge Set Approach to Evaluating Machine Translationcited by 37
5 more
  1. 2019Mining Discourse Markers for Unsupervised Sentence Representation Learningcited by 34
  2. 2020A Set of Recommendations for Assessing Human-Machine Parity in Language Translationcited by 34
  3. 2019Evaluating Pronominal Anaphora in Machine Translation: An Evaluation Measure and a Test Suitecited by 9
  4. 2018Automatic Reference-Based Evaluation of Pronoun Translation Misses the Pointcited by 6
  5. 2019DiscoTK: Using Discourse Structure for Machine Translation Evaluationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 15; 2024: 13; 2025: 18; 2026: 6.

58 papers.

Sort:

By area:

  1. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  2. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
  3. 2023Large language models effectively leverage document-level context for literary translation, but critical errors persistcited by 80
  4. 2022Embarrassingly Easy Document-Level MT Metrics: How to Convert Any Pretrained Metric Into a Document-Level Metriccited by 56
  5. 2022Exploring Document-Level Literary Machine Translation with Parallel Paragraphs from World Literaturecited by 45
  6. 2023Escaping the sentence-level paradigm in machine translationcited by 27
  7. 2024PROXYQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Modelscited by 23
  8. 2024Efficiently Exploring Large Language Models for Document-Level Machine Translation with In-context Learningcited by 23
  9. 2024How Good Are LLMs for Literary Translation, Really? Literary Translation Evaluation with Humans and LLMscited by 23
  10. 2023A Paradigm Shift: The Future of Machine Translation Lies with Large Language Modelscited by 19
  11. 2023Document Flattening: Beyond Concatenating Context for Document-Level Neural Machine Translationcited by 15
  12. 2023Towards Explainable Evaluation Metrics for Machine Translationcited by 15
38 more
  1. 2023Identifying Context-Dependent Translations for Evaluation Set Productioncited by 14
  2. 2023Document-Level Language Models for Machine Translationcited by 12
  3. 2023SLIDE: Reference-free Evaluation for Machine Translation using a Sliding Document Windowcited by 11
  4. 2025AFRIDOC-MT: Document-level MT Corpus for African Languagescited by 11
  5. 2025Multilingual Contextualization of Large Language Models for Document-Level Machine Translationcited by 11
  6. 2024Creative and Context-Aware Translation of East Asian Idioms with GPT-4cited by 10
  7. 2024Watching the Watchers: Exposing Gender Disparities in Machine Translation Quality Estimationcited by 10
  8. 2024Fine-Grained and Multi-Dimensional Metrics for Document-Level Machine Translationcited by 10
  9. 2025LiTransProQA: an LLM-based Literary Translation evaluation metric with Professional Question Answeringcited by 10
  10. 2023Training and Meta-Evaluating Machine Translation Evaluation Metrics at the Paragraph Levelcited by 9
  11. 2025Do LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answeringcited by 9
  12. 2023Improving Long Context Document-Level Machine Translationcited by 8
  13. 2025Bridging the Linguistic Divide: A Survey on Leveraging Large Language Models for Machine Translationcited by 8
  14. 2023Discourse Centric Evaluation of Machine Translation with a Densely Annotated Parallel Corpuscited by 7
  15. 2025Source-primed Multi-turn Conversation Helps Large Language Models Translate Documentscited by 7
  16. 2024DeMPT: Decoding-enhanced Multi-phase Prompt Tuning for Making LLMs Be Better Context-aware Translatorscited by 6
  17. 2024Recovering document annotations for sentence-level bitextcited by 6
  18. 2023Challenges in Context-Aware Neural Machine Translationcited by 5
  19. 2024Is Context Helpful for Chat Translation Evaluation?cited by 5
  20. 2025Two Intermediate Translations Are Better Than One: Fine-tuning LLMs for Document-level Translation Refinementcited by 5
  21. 2024Importance-Aware Data Augmentation for Document-Level Neural Machine Translationcited by 4
  22. 2025Specification-Aware Machine Translation and Evaluation for Purpose Alignmentcited by 4
  23. 2025DITING: A Multi-Agent Evaluation Framework for Benchmarking Web Novel Translationcited by 4
  24. 2022A Bilingual Parallel Corpus with Discourse Annotationscited by 3
  25. 2024Presence or Absence: Are Unknown Word Usages in Dictionaries?cited by 3
  26. 2024VTechAGP: An Academic-to-General-Audience Text Paraphrase Dataset and Benchmark Modelscited by 3
  27. 2025Sci-LoRA: Mixture of Scientific LoRAs for Cross-Domain Lay Paraphrasingcited by 3
  28. 2022Advancing Multilingual Pre-training: TRIP Triangular Document-level Pre-training for Multilingual Language Modelscited by 2
  29. 2023Improving Word Sense Disambiguation in Neural Machine Translation with Salient Document Contextcited by 2
  30. 2024A Context-aware Framework for Translation-mediated Conversationscited by 2
  31. 2025Improving LLM-based Document-level Machine Translation with Multi-Knowledge Fusioncited by 2
  32. 2025CLIRudit: Cross-Lingual Information Retrieval of Scientific Documentscited by 2
  33. 2025Long-context Reference-based MT Quality Estimationcited by 2
  34. 2026Cross-Preference Learning for Sentence-Level and Context-Aware Machine Translationcited by 2
  35. 2025TransBench: Benchmarking Machine Translation for Industrial-Scale Applicationscited by 1
  36. 2025ACADATA: Parallel Dataset of Academic Data for Machine Translationcited by 1
  37. 2026Which Tokens Need Context? A Reference-Based Analysis of Translation Responsibility Using Fertility and Entropycited by 1
  38. 2022Revamping Multilingual Agreement Bidirectionally via Switched Back-translation for Multilingual Neural Machine Translationcited by 0

50 most cited of 58 All 58 (JSON)