# You Are What You Train: Effects of Data Composition on Training Context-aware Machine Translation Models

arXiv: https://arxiv.org/abs/2509.14031 · HTML: https://mlsanity.com/paper/2509.14031/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 56 references that are in the map.
- First posted on arXiv 17 Sep 2025.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [Tower: An Open Multilingual Large Language Model for Translation-Related Tasks](https://mlsanity.com/paper/2402.17733/) (2024)
- [Neural Machine Translation with Extended Context](https://mlsanity.com/paper/1708.05943/) (2017)
- [Evaluating Discourse Phenomena in Neural Machine Translation](https://mlsanity.com/paper/1711.00513/) (2017)
- [Document-Level Neural Machine Translation with Hierarchical Attention Networks](https://mlsanity.com/paper/1809.01576/) (2018)
- [Selective Attention for Context-aware Neural Machine Translation](https://mlsanity.com/paper/1903.08788/) (2019)
- [An Empirical Exploration of Curriculum Learning for Neural Machine Translation](https://mlsanity.com/paper/1811.00739/) (2018)
- [Democratizing Neural Machine Translation with OPUS-MT](https://mlsanity.com/paper/2212.01936/) (2022)
- [Measuring and Increasing Context Usage in Context-Aware Machine Translation](https://mlsanity.com/paper/2105.03482/) (2021)
- [G-Transformer for Document-level Machine Translation](https://mlsanity.com/paper/2105.14761/) (2021)
- [Context-Aware Monolingual Repair for Neural Machine Translation](https://mlsanity.com/paper/1909.01383/) (2019)
- [Does Neural Machine Translation Benefit from Larger Context?](https://mlsanity.com/paper/1704.05135/) (2017)
- [When Does Translation Require Context? A Data-driven, Multilingual Exploration](https://mlsanity.com/paper/2109.07446/) (2021)
- [Long-Short Term Masking Transformer: A Simple but Effective Baseline for Document-level Neural Machine Translation](https://mlsanity.com/paper/2009.09127/) (2020)
- [Escaping the sentence-level paradigm in machine translation](https://mlsanity.com/paper/2304.12959/) (2023)
- [Diving Deep into Context-Aware Neural Machine Translation](https://mlsanity.com/paper/2010.09482/) (2020)
- [Do Context-Aware Translation Models Pay the Right Attention?](https://mlsanity.com/paper/2105.06977/) (2021)
- [Learn To Remember: Transformer with Recurrent Memory for Document-Level Machine Translation](https://mlsanity.com/paper/2205.01546/) (2022)
- [Identifying Context-Dependent Translations for Evaluation Set Production](https://mlsanity.com/paper/2311.02321/) (2023)
- [Divide and Rule: Effective Pre-Training for Context-Aware Multi-Encoder Translation Models](https://mlsanity.com/paper/2103.17151/) (2021)
- [A baseline revisited: Pushing the limits of multi-segment models for context-aware translation](https://mlsanity.com/paper/2210.10906/) (2022)
- [Context Gates for Neural Machine Translation](https://mlsanity.com/paper/1608.06043/) (2016)
- [On Measuring Context Utilization in Document-Level MT Systems](https://mlsanity.com/paper/2402.01404/) (2024)
- [To Label or Not to Label: Hybrid Active Learning for Neural Machine Translation](https://mlsanity.com/paper/2403.09259/) (2024)
- [Pronoun-Targeted Fine-tuning for NMT with Hybrid Losses](https://mlsanity.com/paper/2010.07638/) (2020)
- [Analyzing the Attention Heads for Pronoun Disambiguation in Context-aware Machine Translation Models](https://mlsanity.com/paper/2412.11187/) (2024)

## Built on by (most cited first, 3 of 3)

- [STAR : Sentence Translation Alignment Rate for Document-to-Document Machine Translation](https://mlsanity.com/paper/2608.27161/) (2026)
- [Scaling Attention Head Analysis via Gradient-Based Attribution in Context-Aware Machine Translation](https://mlsanity.com/paper/2609.28117/) (2026)
- [Improving Visual Sensitivity of LLMs on Multimodal Machine Translation with Metric-based Loss Weighting](https://mlsanity.com/paper/2609.31169/) (2026)

## Abstract (arXiv)

Achieving human-level translations requires leveraging context to ensure coherence and handle complex phenomena like pronoun disambiguation. Sparsity of contextually rich examples in the standard training data has been hypothesized as the reason for the difficulty of context utilization. In this work, we systematically validate this claim in both single- and multilingual settings by constructing training datasets with a controlled proportions of contextually relevant examples. We demonstrate a strong association between training data sparsity and model performance confirming sparsity as a key bottleneck. Importantly, we reveal that improvements in one contextual phenomenon do no generalize to others. While we observe some cross-lingual transfer, it is not significantly higher between languages within the same sub-family. Finally, we propose and empirically evaluate two training strategies designed to leverage the available data. These strategies improve context utilization, resulting in accuracy gains of up to 6 and 8 percentage points on the ctxPro evaluation in single- and multilingual settings respectively.