# Can Your Context-Aware MT System Pass the DiP Benchmark Tests? : Evaluation Benchmarks for Discourse Phenomena in Machine Translation

arXiv: https://arxiv.org/abs/2004.14607 · HTML: https://mlsanity.com/paper/2004.14607/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2020.

## Built on by (most cited first, 6 of 6)

- [When Does Translation Require Context? A Data-driven, Multilingual Exploration](https://mlsanity.com/paper/2109.07446/) (2021)
- [A baseline revisited: Pushing the limits of multi-segment models for context-aware translation](https://mlsanity.com/paper/2210.10906/) (2022)
- [DocHPLT: A Massively Multilingual Document-Level Translation Dataset](https://mlsanity.com/paper/2508.13079/) (2025)
- [Revisiting Context Choices for Context-aware Machine Translation](https://mlsanity.com/paper/2109.02995/) (2021)
- [Unlocking Latent Discourse Translation in LLMs Through Quality-Aware Decoding](https://mlsanity.com/paper/2510.06866/) (2025)
- [How Far Can Machine Translation Quality Take You? Extrinsic Discourse Evaluation in Goal-Oriented Setups](https://mlsanity.com/paper/2606.16596/) (2026)

## Abstract (arXiv)

Despite increasing instances of machine translation (MT) systems including contextual information, the evidence for translation quality improvement is sparse, especially for discourse phenomena. Popular metrics like BLEU are not expressive or sensitive enough to capture quality improvements or drops that are minor in size but significant in perception. We introduce the first of their kind MT benchmark datasets that aim to track and hail improvements across four main discourse phenomena: anaphora, lexical consistency, coherence and readability, and discourse connective translation. We also introduce evaluation methods for these tasks, and evaluate several baseline MT systems on the curated datasets. Surprisingly, we find that existing context-aware models do not improve discourse-related translations consistently across languages and phenomena.