# Improving Long Context Document-Level Machine Translation arXiv: https://arxiv.org/abs/2306.05183 ยท HTML: https://mlsanity.com/paper/2306.05183/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 48 of its 67 references that are in the map. - First posted on arXiv 8 Jun 2023. ## Builds on (most cited first, 30 of 48) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014) - [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015) - [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018) - [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019) - [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019) - [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018) - [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020) - [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015) - [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018) - [Adaptive Attention Span in Transformers](https://mlsanity.com/paper/1905.07799/) (2019) - [Context-Aware Neural Machine Translation Learns Anaphora Resolution](https://mlsanity.com/paper/1805.10163/) (2018) - [Neural Machine Translation with Extended Context](https://mlsanity.com/paper/1708.05943/) (2017) - [Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation](https://mlsanity.com/paper/1808.07048/) (2018) - [Evaluating Discourse Phenomena in Neural Machine Translation](https://mlsanity.com/paper/1711.00513/) (2017) - [Document-Level Neural Machine Translation with Hierarchical Attention Networks](https://mlsanity.com/paper/1809.01576/) (2018) - [Improving the Transformer Translation Model with Document-Level Context](https://mlsanity.com/paper/1810.03581/) (2018) - [Selective Attention for Context-aware Neural Machine Translation](https://mlsanity.com/paper/1903.08788/) (2019) - [BlonDe: An Automatic Evaluation Metric for Document-level Machine Translation](https://mlsanity.com/paper/2103.11878/) (2021) - [Microsoft Translator at WMT 2019: Towards Large-Scale Document-Level Neural Machine Translation](https://mlsanity.com/paper/1907.06170/) (2019) - [Exploiting Cross-Sentence Context for Neural Machine Translation](https://mlsanity.com/paper/1704.04347/) (2017) - [Measuring and Increasing Context Usage in Context-Aware Machine Translation](https://mlsanity.com/paper/2105.03482/) (2021) - [G-Transformer for Document-level Machine Translation](https://mlsanity.com/paper/2105.14761/) (2021) - [When and Why is Document-level Context Useful in Neural Machine Translation?](https://mlsanity.com/paper/1910.00294/) (2019) - [Exploring Document-Level Literary Machine Translation with Parallel Paragraphs from World Literature](https://mlsanity.com/paper/2210.14250/) (2022) - [Modeling Localness for Self-Attention Networks](https://mlsanity.com/paper/1810.10182/) (2018) - [Document Context Neural Machine Translation with Memory Networks](https://mlsanity.com/paper/1711.03688/) (2017) - [Learning to Remember Translation History with a Continuous Cache](https://mlsanity.com/paper/1711.09367/) (2017) - [Does Neural Machine Translation Benefit from Larger Context?](https://mlsanity.com/paper/1704.05135/) (2017) - [MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translation](https://mlsanity.com/paper/2211.01355/) (2022) ## Built on by (most cited first, 8 of 8) - [(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Texts](https://mlsanity.com/paper/2405.11804/) (2024) - [Contextual Refinement of Translations: Large Language Models for Sentence and Document-Level Post-Editing](https://mlsanity.com/paper/2310.14855/) (2023) - [Document-Level Language Models for Machine Translation](https://mlsanity.com/paper/2310.12303/) (2023) - [AFRIDOC-MT: Document-level MT Corpus for African Languages](https://mlsanity.com/paper/2501.06374/) (2025) - [Investigating Length Issues in Document-level Machine Translation](https://mlsanity.com/paper/2412.17592/) (2024) - [SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimization](https://mlsanity.com/paper/2505.11166/) (2025) - [Extending Automatic Machine Translation Evaluation to Book-Length Documents](https://mlsanity.com/paper/2509.17249/) (2025) - [XToM: Exploring the Multilingual Theory of Mind for Large Language Models](https://mlsanity.com/paper/2506.02461/) (2025) ## Abstract (arXiv) Document-level context for neural machine translation (NMT) is crucial to improve the translation consistency and cohesion, the translation of ambiguous inputs, as well as several other linguistic phenomena. Many works have been published on the topic of document-level NMT, but most restrict the system to only local context, typically including just the one or two preceding sentences as additional information. This might be enough to resolve some ambiguous inputs, but it is probably not sufficient to capture some document-level information like the topic or style of a conversation. When increasing the context size beyond just the local context, there are two challenges: (i) the~memory usage increases exponentially (ii) the translation performance starts to degrade. We argue that the widely-used attention mechanism is responsible for both issues. Therefore, we propose a constrained attention variant that focuses the attention on the most relevant parts of the sequence, while simultaneously reducing the memory consumption. For evaluation, we utilize targeted test sets in combination with novel evaluation techniques to analyze the translations in regards to specific discourse-related phenomena. We find that our approach is a good compromise between sentence-level NMT vs attending to the full context, especially in low resource scenarios.