# Discourse Structure in Machine Translation Evaluation

arXiv: https://arxiv.org/abs/1710.01504 · HTML: https://mlsanity.com/paper/1710.01504/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Oct 2017.

## Built on by (most cited first, 7 of 7)

- [DiscoScore: Evaluating Text Generation with BERT and Discourse Coherence](https://mlsanity.com/paper/2201.11176/) (2022)
- [Towards Explainable Evaluation Metrics for Natural Language Generation](https://mlsanity.com/paper/2203.11131/) (2022)
- [RST Parsing from Scratch](https://mlsanity.com/paper/2105.10861/) (2021)
- [Do LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answering](https://mlsanity.com/paper/2504.07583/) (2025)
- [Discourse-Driven Evaluation: Unveiling Factual Inconsistency in Long Document Summarization](https://mlsanity.com/paper/2502.06185/) (2025)
- [Can we obtain significant success in RST discourse parsing by using Large Language Models?](https://mlsanity.com/paper/2403.05065/) (2024)
- [When Looks Do Not Lie: Discourse Structure Guided In-Context Learning for Faithful Diagram Generation](https://mlsanity.com/paper/2601.20476/) (2026)

## Abstract (arXiv)

In this article, we explore the potential of using sentence-level discourse structure for machine translation evaluation. We first design discourse-aware similarity measures, which use all-subtree kernels to compare discourse parse trees in accordance with the Rhetorical Structure Theory (RST). Then, we show that a simple linear combination with these measures can help improve various existing machine translation evaluation metrics regarding correlation with human judgments both at the segment- and at the system-level. This suggests that discourse information is complementary to the information used by many of the existing evaluation metrics, and thus it could be taken into account when developing richer evaluation metrics, such as the WMT-14 winning combined metric DiscoTKparty. We also provide a detailed analysis of the relevance of various discourse elements and relations from the RST parse trees for machine translation evaluation. In particular we show that: (i) all aspects of the RST tree are relevant, (ii) nuclearity is more useful than relation type, and (iii) the similarity of the translation RST tree to the reference tree is positively correlated with translation quality.