# Does Multi-Encoder Help? A Case Study on Context-Aware Neural Machine Translation

arXiv: https://arxiv.org/abs/2005.03393 · HTML: https://mlsanity.com/paper/2005.03393/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 May 2020.

## Built on by (most cited first, 30 of 34)

- [On Vision Features in Multimodal Machine Translation](https://mlsanity.com/paper/2203.09173/) (2022)
- [Gender Neutralization for an Inclusive Machine Translation: from Theoretical Foundations to Open Challenges](https://mlsanity.com/paper/2301.10075/) (2023)
- [Do Context-Aware Translation Models Pay the Right Attention?](https://mlsanity.com/paper/2105.06977/) (2021)
- [Modeling Bilingual Conversational Characteristics for Neural Chat Translation](https://mlsanity.com/paper/2107.11164/) (2021)
- [Assemble Foundation Models for Automatic Code Summarization](https://mlsanity.com/paper/2201.05222/) (2022)
- [Diverse Pretrained Context Encodings Improve Document Translation](https://mlsanity.com/paper/2106.03717/) (2021)
- [Divide and Rule: Effective Pre-Training for Context-Aware Multi-Encoder Translation Models](https://mlsanity.com/paper/2103.17151/) (2021)
- [Towards Making the Most of Dialogue Characteristics for Neural Chat Translation](https://mlsanity.com/paper/2109.00668/) (2021)
- [A baseline revisited: Pushing the limits of multi-segment models for context-aware translation](https://mlsanity.com/paper/2210.10906/) (2022)
- [Contrastive Learning for Context-aware Neural Machine TranslationUsing Coreference Information](https://mlsanity.com/paper/2109.05712/) (2021)
- [On Measuring Context Utilization in Document-Level MT Systems](https://mlsanity.com/paper/2402.01404/) (2024)
- [Improving Long Context Document-Level Machine Translation](https://mlsanity.com/paper/2306.05183/) (2023)
- [Focused Concatenation for Context-Aware Neural Machine Translation](https://mlsanity.com/paper/2210.13388/) (2022)
- [Multi-Encoder Learning and Stream Fusion for Transformer-Based End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2104.00120/) (2021)
- [P-Transformer: Towards Better Document-to-Document Neural Machine Translation](https://mlsanity.com/paper/2212.05830/) (2022)
- [Text revision in Scientific Writing Assistance: An Overview](https://mlsanity.com/paper/2303.16726/) (2023)
- [Challenges in Context-Aware Neural Machine Translation](https://mlsanity.com/paper/2305.13751/) (2023)
- [Towards Personalised and Document-level Machine Translation of Dialogue](https://mlsanity.com/paper/2102.10979/) (2021)
- [A Case Study on Context-Aware Neural Machine Translation with Multi-Task Learning](https://mlsanity.com/paper/2407.03076/) (2024)
- [A Multi-task Multi-stage Transitional Training Framework for Neural Chat Translation](https://mlsanity.com/paper/2301.11749/) (2023)
- [Incorporating Probing Signals into Multimodal Machine Translation via Visual Question-Answering Pairs](https://mlsanity.com/paper/2310.17133/) (2023)
- [Enhancing AI-based Generation of Software Exploits with Contextual Information](https://mlsanity.com/paper/2408.02402/) (2024)
- [Impact of Visual Context on Noisy Multimodal NMT: An Empirical Study for English to Indian Languages](https://mlsanity.com/paper/2308.16075/) (2023)
- [Improving Word Sense Disambiguation in Neural Machine Translation with Salient Document Context](https://mlsanity.com/paper/2311.15507/) (2023)
- [Sequence Shortening for Context-Aware Machine Translation](https://mlsanity.com/paper/2402.01416/) (2024)
- [Revisiting Context Choices for Context-aware Machine Translation](https://mlsanity.com/paper/2109.02995/) (2021)
- [A Case Study on Context Encoding in Multi-Encoder based Document-Level Neural Machine Translation](https://mlsanity.com/paper/2308.06063/) (2023)
- [OpenWHO: A Document-Level Parallel Corpus for Health Translation in Low-Resource Languages](https://mlsanity.com/paper/2508.16048/) (2025)
- [Which Tokens Need Context? A Reference-Based Analysis of Translation Responsibility Using Fertility and Entropy](https://mlsanity.com/paper/2606.29489/) (2026)
- [On Search Strategies for Document-Level Neural Machine Translation](https://mlsanity.com/paper/2306.05116/) (2023)

## Abstract (arXiv)

In encoder-decoder neural models, multiple encoders are in general used to represent the contextual information in addition to the individual sentence. In this paper, we investigate multi-encoder approaches in documentlevel neural machine translation (NMT). Surprisingly, we find that the context encoder does not only encode the surrounding sentences but also behaves as a noise generator. This makes us rethink the real benefits of multi-encoder in context-aware translation - some of the improvements come from robust training. We compare several methods that introduce noise and/or well-tuned dropout setup into the training of these encoders. Experimental results show that noisy training plays an important role in multi-encoder-based NMT, especially when the training data is small. Also, we establish a new state-of-the-art on IWSLT Fr-En task by careful use of noise generation and dropout methods.