# Evaluating Discourse Phenomena in Neural Machine Translation

arXiv: https://arxiv.org/abs/1711.00513 · HTML: https://mlsanity.com/paper/1711.00513/

## Key facts

- Cited by 87 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Nov 2017.

## Built on by (most cited first, 30 of 87)

- [Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs](https://mlsanity.com/paper/2311.09469/) (2023)
- [Document-Level Machine Translation with Large Language Models](https://mlsanity.com/paper/2304.02210/) (2023)
- [Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM](https://mlsanity.com/paper/2303.01911/) (2023)
- [Adapting Large Language Models for Document-Level Machine Translation](https://mlsanity.com/paper/2401.06468/) (2024)
- [BlonDe: An Automatic Evaluation Metric for Document-level Machine Translation](https://mlsanity.com/paper/2103.11878/) (2021)
- [Embarrassingly Easy Document-Level MT Metrics: How to Convert Any Pretrained Metric Into a Document-Level Metric](https://mlsanity.com/paper/2209.13654/) (2022)
- [Measuring and Increasing Context Usage in Context-Aware Machine Translation](https://mlsanity.com/paper/2105.03482/) (2021)
- [Exploring Document-Level Literary Machine Translation with Parallel Paragraphs from World Literature](https://mlsanity.com/paper/2210.14250/) (2022)
- [When Does Translation Require Context? A Data-driven, Multilingual Exploration](https://mlsanity.com/paper/2109.07446/) (2021)
- [DiscoScore: Evaluating Text Generation with BERT and Discourse Coherence](https://mlsanity.com/paper/2201.11176/) (2022)
- [Escaping the sentence-level paradigm in machine translation](https://mlsanity.com/paper/2304.12959/) (2023)
- [DelTA: An Online Document-Level Translation Agent Based on Multi-Level Memory](https://mlsanity.com/paper/2410.08143/) (2024)
- [RuCoLA: Russian Corpus of Linguistic Acceptability](https://mlsanity.com/paper/2210.12814/) (2022)
- [Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluation](https://mlsanity.com/paper/2212.10140/) (2022)
- [NoiseQA: Challenge Set Evaluation for User-Centric Question Answering](https://mlsanity.com/paper/2102.08345/) (2021)
- [Considerations for meaningful sign language machine translation based on glosses](https://mlsanity.com/paper/2211.15464/) (2022)
- [Gender Neutralization for an Inclusive Machine Translation: from Theoretical Foundations to Open Challenges](https://mlsanity.com/paper/2301.10075/) (2023)
- [Do Context-Aware Translation Models Pay the Right Attention?](https://mlsanity.com/paper/2105.06977/) (2021)
- [GigaST: A 10,000-hour Pseudo Speech Translation Corpus](https://mlsanity.com/paper/2204.03939/) (2022)
- [Modeling Bilingual Conversational Characteristics for Neural Chat Translation](https://mlsanity.com/paper/2107.11164/) (2021)
- [Interactive-Chain-Prompting: Ambiguity Resolution for Crosslingual Conditional Generation with Interaction](https://mlsanity.com/paper/2301.10309/) (2023)
- [On the Limits of Minimal Pairs in Contrastive Evaluation](https://mlsanity.com/paper/2109.07465/) (2021)
- [Document Flattening: Beyond Concatenating Context for Document-Level Neural Machine Translation](https://mlsanity.com/paper/2302.08079/) (2023)
- [Identifying Context-Dependent Translations for Evaluation Set Production](https://mlsanity.com/paper/2311.02321/) (2023)
- [Diverse Pretrained Context Encodings Improve Document Translation](https://mlsanity.com/paper/2106.03717/) (2021)
- [Enhancing Document-level Translation of Large Language Model via Translation Mixed-instructions](https://mlsanity.com/paper/2401.08088/) (2024)
- [Divide and Rule: Effective Pre-Training for Context-Aware Multi-Encoder Translation Models](https://mlsanity.com/paper/2103.17151/) (2021)
- [Towards Making the Most of Dialogue Characteristics for Neural Chat Translation](https://mlsanity.com/paper/2109.00668/) (2021)
- [Document-Level Language Models for Machine Translation](https://mlsanity.com/paper/2310.12303/) (2023)
- [SLIDE: Reference-free Evaluation for Machine Translation using a Sliding Document Window](https://mlsanity.com/paper/2309.08832/) (2023)

## Abstract (arXiv)

For machine translation to tackle discourse phenomena, models must have access to extra-sentential linguistic context. There has been recent interest in modelling context in neural machine translation (NMT), but models have been principally evaluated with standard automatic metrics, poorly adapted to evaluating discourse phenomena. In this article, we present hand-crafted, discourse test sets, designed to test the models' ability to exploit previous source and target sentences. We investigate the performance of recently proposed multi-encoder NMT models trained on subtitles for English to French. We also explore a novel way of exploiting context from the previous sentence. Despite gains using BLEU, multi-encoder models give limited improvement in the handling of discourse phenomena: 50% accuracy on our coreference test set and 53.5% for coherence/cohesion (compared to a non-contextual baseline of 50%). A simple strategy of decoding the concatenation of the previous and current sentence leads to good performance, and our novel strategy of multi-encoding and decoding of two sentences leads to the best performance (72.5% for coreference and 57% for coherence/cohesion), highlighting the importance of target-side context.