# Exploring Paracrawl for Document-level Neural Machine Translation

arXiv: https://arxiv.org/abs/2304.10216 · HTML: https://mlsanity.com/paper/2304.10216/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 28 references that are in the map.
- First posted on arXiv 20 Apr 2023.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Context-Aware Neural Machine Translation Learns Anaphora Resolution](https://mlsanity.com/paper/1805.10163/) (2018)
- [Evaluating Discourse Phenomena in Neural Machine Translation](https://mlsanity.com/paper/1711.00513/) (2017)
- [Improving the Transformer Translation Model with Document-Level Context](https://mlsanity.com/paper/1810.03581/) (2018)
- [Selective Attention for Context-aware Neural Machine Translation](https://mlsanity.com/paper/1903.08788/) (2019)
- [Measuring and Increasing Context Usage in Context-Aware Machine Translation](https://mlsanity.com/paper/2105.03482/) (2021)
- [G-Transformer for Document-level Machine Translation](https://mlsanity.com/paper/2105.14761/) (2021)
- [Document Context Neural Machine Translation with Memory Networks](https://mlsanity.com/paper/1711.03688/) (2017)
- [Learning to Remember Translation History with a Continuous Cache](https://mlsanity.com/paper/1711.09367/) (2017)
- [Dynamic Context Selection for Document-level Neural Machine Translation via Reinforcement Learning](https://mlsanity.com/paper/2010.04314/) (2020)
- [Diverse Pretrained Context Encodings Improve Document Translation](https://mlsanity.com/paper/2106.03717/) (2021)
- [Document Graph for Neural Machine Translation](https://mlsanity.com/paper/2012.03477/) (2020)
- [Enhancing Context Modeling with a Query-Guided Capsule Network for Document-level Translation](https://mlsanity.com/paper/1909.00564/) (2019)
- [Using Context in Neural Machine Translation Training Objectives](https://mlsanity.com/paper/2005.01483/) (2020)
- [Fusing Recency into Neural Machine Translation with an Inter-Sentence Gate Model](https://mlsanity.com/paper/1806.04466/) (2018)
- [Leveraging Discourse Rewards for Document-Level Neural Machine Translation](https://mlsanity.com/paper/2010.03732/) (2020)

## Built on by (most cited first, 2 of 2)

- [SPEED: Speculative Pipelined Execution for Efficient Decoding](https://mlsanity.com/paper/2310.12072/) (2023)
- [Towards Chapter-to-Chapter Context-Aware Literary Translation via Large Language Models](https://mlsanity.com/paper/2407.08978/) (2024)

## Abstract (arXiv)

Document-level neural machine translation (NMT) has outperformed sentence-level NMT on a number of datasets. However, document-level NMT is still not widely adopted in real-world translation systems mainly due to the lack of large-scale general-domain training data for document-level NMT. We examine the effectiveness of using Paracrawl for learning document-level translation. Paracrawl is a large-scale parallel corpus crawled from the Internet and contains data from various domains. The official Paracrawl corpus was released as parallel sentences (extracted from parallel webpages) and therefore previous works only used Paracrawl for learning sentence-level translation. In this work, we extract parallel paragraphs from Paracrawl parallel webpages using automatic sentence alignments and we use the extracted parallel paragraphs as parallel documents for training document-level translation models. We show that document-level NMT models trained with only parallel paragraphs from Paracrawl can be used to translate real documents from TED, News and Europarl, outperforming sentence-level NMT models. We also perform a targeted pronoun evaluation and show that document-level models trained with Paracrawl data can help context-aware pronoun translation.