# Measuring and Increasing Context Usage in Context-Aware Machine Translation

arXiv: https://arxiv.org/abs/2105.03482 · HTML: https://mlsanity.com/paper/2105.03482/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 31 references that are in the map.
- First posted on arXiv 7 May 2021.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Achieving Human Parity on Automatic Chinese to English News Translation](https://mlsanity.com/paper/1803.05567/) (2018)
- [Neural Machine Translation with Extended Context](https://mlsanity.com/paper/1708.05943/) (2017)
- [Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation](https://mlsanity.com/paper/1808.07048/) (2018)
- [Evaluating Discourse Phenomena in Neural Machine Translation](https://mlsanity.com/paper/1711.00513/) (2017)
- [Document-Level Neural Machine Translation with Hierarchical Attention Networks](https://mlsanity.com/paper/1809.01576/) (2018)
- [Edinburgh Neural Machine Translation Systems for WMT 16](https://mlsanity.com/paper/1606.02891/) (2016)
- [Improving the Transformer Translation Model with Document-Level Context](https://mlsanity.com/paper/1810.03581/) (2018)
- [Selective Attention for Context-aware Neural Machine Translation](https://mlsanity.com/paper/1903.08788/) (2019)
- [Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation](https://mlsanity.com/paper/1808.10432/) (2018)
- [Exploiting Cross-Sentence Context for Neural Machine Translation](https://mlsanity.com/paper/1704.04347/) (2017)
- [When and Why is Document-level Context Useful in Neural Machine Translation?](https://mlsanity.com/paper/1910.00294/) (2019)
- [Document Context Neural Machine Translation with Memory Networks](https://mlsanity.com/paper/1711.03688/) (2017)
- [Learning to Remember Translation History with a Continuous Cache](https://mlsanity.com/paper/1711.09367/) (2017)
- [Does Neural Machine Translation Benefit from Larger Context?](https://mlsanity.com/paper/1704.05135/) (2017)
- [It's Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Information](https://mlsanity.com/paper/2005.02354/) (2020)
- [A Survey on Document-level Neural Machine Translation: Methods and Evaluation](https://mlsanity.com/paper/1912.08494/) (2019)
- [Modeling Discourse Structure for Document-level Neural Machine Translation](https://mlsanity.com/paper/2006.04721/) (2020)
- [Context-Aware Learning for Neural Machine Translation](https://mlsanity.com/paper/1903.04715/) (2019)

## Built on by (most cited first, 30 of 48)

- [Learning to Filter Context for Retrieval-Augmented Generation](https://mlsanity.com/paper/2311.08377/) (2023)
- [Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformers](https://mlsanity.com/paper/2109.04448/) (2021)
- [Embarrassingly Easy Document-Level MT Metrics: How to Convert Any Pretrained Metric Into a Document-Level Metric](https://mlsanity.com/paper/2209.13654/) (2022)
- [When Does Translation Require Context? A Data-driven, Multilingual Exploration](https://mlsanity.com/paper/2109.07446/) (2021)
- [HalOmi: A Manually Annotated Benchmark for Multilingual Hallucination and Omission Detection in Machine Translation](https://mlsanity.com/paper/2305.11746/) (2023)
- [DelTA: An Online Document-Level Translation Agent Based on Multi-Level Memory](https://mlsanity.com/paper/2410.08143/) (2024)
- [A Bayesian Framework for Information-Theoretic Probing](https://mlsanity.com/paper/2109.03853/) (2021)
- [Vision Matters When It Should: Sanity Checking Multimodal Machine Translation Models](https://mlsanity.com/paper/2109.03415/) (2021)
- [How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?](https://mlsanity.com/paper/2412.18495/) (2024)
- [On the Limits of Minimal Pairs in Contrastive Evaluation](https://mlsanity.com/paper/2109.07465/) (2021)
- [Document Flattening: Beyond Concatenating Context for Document-Level Neural Machine Translation](https://mlsanity.com/paper/2302.08079/) (2023)
- [Predicting Attention Sparsity in Transformers](https://mlsanity.com/paper/2109.12188/) (2021)
- [A baseline revisited: Pushing the limits of multi-segment models for context-aware translation](https://mlsanity.com/paper/2210.10906/) (2022)
- [Multilingual Contextualization of Large Language Models for Document-Level Machine Translation](https://mlsanity.com/paper/2504.12140/) (2025)
- [Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspective](https://mlsanity.com/paper/2210.08478/) (2022)
- [On Measuring Context Utilization in Document-Level MT Systems](https://mlsanity.com/paper/2402.01404/) (2024)
- [Improving Long Context Document-Level Machine Translation](https://mlsanity.com/paper/2306.05183/) (2023)
- [Quantifying the Plausibility of Context Reliance in Neural Machine Translation](https://mlsanity.com/paper/2310.01188/) (2023)
- [Encoding Sentence Position in Context-Aware Neural Machine Translation with Concatenation](https://mlsanity.com/paper/2302.06459/) (2023)
- [How Effective are State Space Models for Machine Translation?](https://mlsanity.com/paper/2407.05489/) (2024)
- [Towards Unsupervised Recognition of Token-level Semantic Differences in Related Documents](https://mlsanity.com/paper/2305.13303/) (2023)
- [Challenges in Context-Aware Neural Machine Translation](https://mlsanity.com/paper/2305.13751/) (2023)
- [Is Context Helpful for Chat Translation Evaluation?](https://mlsanity.com/paper/2403.08314/) (2024)
- [Importance-Aware Data Augmentation for Document-Level Neural Machine Translation](https://mlsanity.com/paper/2401.15360/) (2024)
- [M3T: A New Benchmark Dataset for Multi-Modal Document-Level Machine Translation](https://mlsanity.com/paper/2406.08255/) (2024)
- [Analyzing the Attention Heads for Pronoun Disambiguation in Context-aware Machine Translation Models](https://mlsanity.com/paper/2412.11187/) (2024)
- [Long-form Simultaneous Speech Translation: Thesis Proposal](https://mlsanity.com/paper/2310.11141/) (2023)
- [You Are What You Train: Effects of Data Composition on Training Context-aware Machine Translation Models](https://mlsanity.com/paper/2509.14031/) (2025)
- [Exploring Paracrawl for Document-level Neural Machine Translation](https://mlsanity.com/paper/2304.10216/) (2023)
- [Improving Word Sense Disambiguation in Neural Machine Translation with Salient Document Context](https://mlsanity.com/paper/2311.15507/) (2023)

## Abstract (arXiv)

Recent work in neural machine translation has demonstrated both the necessity and feasibility of using inter-sentential context -- context from sentences other than those currently being translated. However, while many current methods present model architectures that theoretically can use this extra context, it is often not clear how much they do actually utilize it at translation time. In this paper, we introduce a new metric, conditional cross-mutual information, to quantify the usage of context by these models. Using this metric, we measure how much document-level machine translation systems use particular varieties of context. We find that target context is referenced more than source context, and that conditioning on a longer context has a diminishing effect on results. We then introduce a new, simple training method, context-aware word dropout, to increase the usage of context by context-aware models. Experiments show that our method increases context usage and that this reflects on the translation quality according to metrics such as BLEU and COMET, as well as performance on anaphoric pronoun resolution and lexical cohesion contrastive datasets.