# A Case Study on Context Encoding in Multi-Encoder based Document-Level Neural Machine Translation

arXiv: https://arxiv.org/abs/2308.06063 · HTML: https://mlsanity.com/paper/2308.06063/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 49 references that are in the map.
- First posted on arXiv 11 Aug 2023.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Context-Aware Neural Machine Translation Learns Anaphora Resolution](https://mlsanity.com/paper/1805.10163/) (2018)
- [Neural Machine Translation with Extended Context](https://mlsanity.com/paper/1708.05943/) (2017)
- [Multi-task Sequence to Sequence Learning](https://mlsanity.com/paper/1511.06114/) (2015)
- [Evaluating Discourse Phenomena in Neural Machine Translation](https://mlsanity.com/paper/1711.00513/) (2017)
- [Training Tips for the Transformer Model](https://mlsanity.com/paper/1804.00247/) (2018)
- [Document-Level Neural Machine Translation with Hierarchical Attention Networks](https://mlsanity.com/paper/1809.01576/) (2018)
- [Improving the Transformer Translation Model with Document-Level Context](https://mlsanity.com/paper/1810.03581/) (2018)
- [Selective Attention for Context-aware Neural Machine Translation](https://mlsanity.com/paper/1903.08788/) (2019)
- [Microsoft Translator at WMT 2019: Towards Large-Scale Document-Level Neural Machine Translation](https://mlsanity.com/paper/1907.06170/) (2019)
- [Tied Multitask Learning for Neural Speech Translation](https://mlsanity.com/paper/1802.06655/) (2018)
- [G-Transformer for Document-level Machine Translation](https://mlsanity.com/paper/2105.14761/) (2021)
- [When and Why is Document-level Context Useful in Neural Machine Translation?](https://mlsanity.com/paper/1910.00294/) (2019)
- [Document Context Neural Machine Translation with Memory Networks](https://mlsanity.com/paper/1711.03688/) (2017)
- [Does Multi-Encoder Help? A Case Study on Context-Aware Neural Machine Translation](https://mlsanity.com/paper/2005.03393/) (2020)
- [Rethinking Document-level Neural Machine Translation](https://mlsanity.com/paper/2010.08961/) (2020)
- [Neural Machine Translation with Reconstruction](https://mlsanity.com/paper/1611.01874/) (2016)
- [Long-Short Term Masking Transformer: A Simple but Effective Baseline for Document-level Neural Machine Translation](https://mlsanity.com/paper/2009.09127/) (2020)
- [Escaping the sentence-level paradigm in machine translation](https://mlsanity.com/paper/2304.12959/) (2023)
- [Dynamic Context Selection for Document-level Neural Machine Translation via Reinforcement Learning](https://mlsanity.com/paper/2010.04314/) (2020)
- [Diving Deep into Context-Aware Neural Machine Translation](https://mlsanity.com/paper/2010.09482/) (2020)
- [Input Combination Strategies for Multi-Source Transformer Decoder](https://mlsanity.com/paper/1811.04716/) (2018)

## Built on by (most cited first, 1 of 1)

- [Impact of Visual Context on Noisy Multimodal NMT: An Empirical Study for English to Indian Languages](https://mlsanity.com/paper/2308.16075/) (2023)

## Abstract (arXiv)

Recent studies have shown that the multi-encoder models are agnostic to the choice of context, and the context encoder generates noise which helps improve the models in terms of BLEU score. In this paper, we further explore this idea by evaluating with context-aware pronoun translation test set by training multi-encoder models trained on three different context settings viz, previous two sentences, random two sentences, and a mix of both as context. Specifically, we evaluate the models on the ContraPro test set to study how different contexts affect pronoun translation accuracy. The results show that the model can perform well on the ContraPro test set even when the context is random. We also analyze the source representations to study whether the context encoder generates noise. Our analysis shows that the context encoder provides sufficient information to learn discourse-level information. Additionally, we observe that mixing the selected context (the previous two sentences in this case) and the random context is generally better than the other settings.