# Microsoft Translator at WMT 2019: Towards Large-Scale Document-Level Neural Machine Translation

arXiv: https://arxiv.org/abs/1907.06170 · HTML: https://mlsanity.com/paper/1907.06170/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jul 2019.

## Built on by (most cited first, 30 of 56)

- [Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense](https://mlsanity.com/paper/2303.13408/) (2023)
- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://mlsanity.com/paper/2103.12028/) (2021)
- [Large language models effectively leverage document-level context for literary translation, but critical errors persist](https://mlsanity.com/paper/2304.03245/) (2023)
- [Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey](https://mlsanity.com/paper/2104.06951/) (2021)
- [G-Transformer for Document-level Machine Translation](https://mlsanity.com/paper/2105.14761/) (2021)
- [Exploring Document-Level Literary Machine Translation with Parallel Paragraphs from World Literature](https://mlsanity.com/paper/2210.14250/) (2022)
- [Escaping the sentence-level paradigm in machine translation](https://mlsanity.com/paper/2304.12959/) (2023)
- [Ethical Considerations for Machine Translation of Indigenous Languages: Giving a Voice to the Speakers](https://mlsanity.com/paper/2305.19474/) (2023)
- [Nugget: Neural Agglomerative Embeddings of Text](https://mlsanity.com/paper/2310.01732/) (2023)
- [Learn To Remember: Transformer with Recurrent Memory for Document-Level Machine Translation](https://mlsanity.com/paper/2205.01546/) (2022)
- [Document Flattening: Beyond Concatenating Context for Document-Level Neural Machine Translation](https://mlsanity.com/paper/2302.08079/) (2023)
- [Identifying Context-Dependent Translations for Evaluation Set Production](https://mlsanity.com/paper/2311.02321/) (2023)
- [Diverse Pretrained Context Encodings Improve Document Translation](https://mlsanity.com/paper/2106.03717/) (2021)
- [Enhancing Document-level Translation of Large Language Model via Translation Mixed-instructions](https://mlsanity.com/paper/2401.08088/) (2024)
- [Divide and Rule: Effective Pre-Training for Context-Aware Multi-Encoder Translation Models](https://mlsanity.com/paper/2103.17151/) (2021)
- [Document-Level Language Models for Machine Translation](https://mlsanity.com/paper/2310.12303/) (2023)
- [Data Augmentation by Concatenation for Low-Resource Translation: A Mystery and a Solution](https://mlsanity.com/paper/2105.01691/) (2021)
- [A baseline revisited: Pushing the limits of multi-segment models for context-aware translation](https://mlsanity.com/paper/2210.10906/) (2022)
- [Data Augmentation for Low-Resource Named Entity Recognition Using Backtranslation](https://mlsanity.com/paper/2108.11703/) (2021)
- [Critical Learning Periods: Leveraging Early Training Dynamics for Efficient Data Pruning](https://mlsanity.com/paper/2405.19462/) (2024)
- [Investigating Length Issues in Document-level Machine Translation](https://mlsanity.com/paper/2412.17592/) (2024)
- [From Simultaneous to Streaming Machine Translation by Leveraging Streaming History](https://mlsanity.com/paper/2203.02459/) (2022)
- [DOCmT5: Document-Level Pretraining of Multilingual Language Models](https://mlsanity.com/paper/2112.08709/) (2021)
- [Improving Long Context Document-Level Machine Translation](https://mlsanity.com/paper/2306.05183/) (2023)
- [A Comparison of Approaches to Document-level Machine Translation](https://mlsanity.com/paper/2101.11040/) (2021)
- [Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort](https://mlsanity.com/paper/2105.13385/) (2021)
- [Multilingual Document-Level Translation Enables Zero-Shot Transfer From Sentences to Documents](https://mlsanity.com/paper/2109.10341/) (2021)
- [Focused Concatenation for Context-Aware Neural Machine Translation](https://mlsanity.com/paper/2210.13388/) (2022)
- [MultiClaimNet: A Massively Multilingual Dataset of Fact-Checked Claim Clusters](https://mlsanity.com/paper/2503.22280/) (2025)
- [Encoding Sentence Position in Context-Aware Neural Machine Translation with Concatenation](https://mlsanity.com/paper/2302.06459/) (2023)

## Abstract (arXiv)

This paper describes the Microsoft Translator submissions to the WMT19 news translation shared task for English-German. Our main focus is document-level neural machine translation with deep transformer models. We start with strong sentence-level baselines, trained on large-scale data created via data-filtering and noisy back-translation and find that back-translation seems to mainly help with translationese input. We explore fine-tuning techniques, deeper models and different ensembling strategies to counter these effects. Using document boundaries present in the authentic and synthetic parallel data, we create sequences of up to 1000 subword segments and train transformer translation models. We experiment with data augmentation techniques for the smaller authentic data with document-boundaries and for larger authentic data without boundaries. We further explore multi-task training for the incorporation of document-level source language monolingual data via the BERT-objective on the encoder and two-pass decoding for combinations of sentence-level and document-level systems. Based on preliminary human evaluation results, evaluators strongly prefer the document-level systems over our comparable sentence-level system. The document-level systems also seem to score higher than the human references in source-based direct assessment.