# A baseline revisited: Pushing the limits of multi-segment models for context-aware translation arXiv: https://arxiv.org/abs/2210.10906 ยท HTML: https://mlsanity.com/paper/2210.10906/ ## Key facts - Cited by 11 papers in the map (data through 14 Oct 2026). - Builds on 30 of its 42 references that are in the map. - First posted on arXiv 19 Oct 2022. ## Builds on (most cited first, 30 of 30) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015) - [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020) - [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015) - [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016) - [Understanding Knowledge Distillation in Non-autoregressive Machine Translation](https://mlsanity.com/paper/1911.02727/) (2019) - [Achieving Human Parity on Automatic Chinese to English News Translation](https://mlsanity.com/paper/1803.05567/) (2018) - [Neural Machine Translation with Extended Context](https://mlsanity.com/paper/1708.05943/) (2017) - [Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation](https://mlsanity.com/paper/1808.07048/) (2018) - [Document-Level Neural Machine Translation with Hierarchical Attention Networks](https://mlsanity.com/paper/1809.01576/) (2018) - [Selective Attention for Context-aware Neural Machine Translation](https://mlsanity.com/paper/1903.08788/) (2019) - [Microsoft Translator at WMT 2019: Towards Large-Scale Document-Level Neural Machine Translation](https://mlsanity.com/paper/1907.06170/) (2019) - [Measuring and Increasing Context Usage in Context-Aware Machine Translation](https://mlsanity.com/paper/2105.03482/) (2021) - [G-Transformer for Document-level Machine Translation](https://mlsanity.com/paper/2105.14761/) (2021) - [Context-Aware Monolingual Repair for Neural Machine Translation](https://mlsanity.com/paper/1909.01383/) (2019) - [When Does Translation Require Context? A Data-driven, Multilingual Exploration](https://mlsanity.com/paper/2109.07446/) (2021) - [Does Multi-Encoder Help? A Case Study on Context-Aware Neural Machine Translation](https://mlsanity.com/paper/2005.03393/) (2020) - [Diving Deep into Context-Aware Neural Machine Translation](https://mlsanity.com/paper/2010.09482/) (2020) - [Do Context-Aware Translation Models Pay the Right Attention?](https://mlsanity.com/paper/2105.06977/) (2021) - [Explaining Sequence-Level Knowledge Distillation as Data-Augmentation for Neural Machine Translation](https://mlsanity.com/paper/1912.03334/) (2019) - [The Sockeye 2 Neural Machine Translation Toolkit at AMTA 2020](https://mlsanity.com/paper/2008.04885/) (2020) - [On the Limits of Minimal Pairs in Contrastive Evaluation](https://mlsanity.com/paper/2109.07465/) (2021) - [Diverse Pretrained Context Encodings Improve Document Translation](https://mlsanity.com/paper/2106.03717/) (2021) - [A Survey on Document-level Neural Machine Translation: Methods and Evaluation](https://mlsanity.com/paper/1912.08494/) (2019) - [How Does Distilled Data Complexity Impact the Quality and Confidence of Non-Autoregressive Machine Translation?](https://mlsanity.com/paper/2105.12900/) (2021) - [Data Augmentation by Concatenation for Low-Resource Translation: A Mystery and a Solution](https://mlsanity.com/paper/2105.01691/) (2021) - [Evaluating Pronominal Anaphora in Machine Translation: An Evaluation Measure and a Test Suite](https://mlsanity.com/paper/1909.00131/) (2019) - [Can Your Context-Aware MT System Pass the DiP Benchmark Tests? : Evaluation Benchmarks for Discourse Phenomena in Machine Translation](https://mlsanity.com/paper/2004.14607/) (2020) - [Pronoun-Targeted Fine-tuning for NMT with Hybrid Losses](https://mlsanity.com/paper/2010.07638/) (2020) - [On User Interfaces for Large-Scale Document-Level Human Evaluation of Machine Translation Outputs](https://mlsanity.com/paper/2104.10408/) (2021) ## Built on by (most cited first, 11 of 11) - [MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translation](https://mlsanity.com/paper/2211.01355/) (2022) - [Improving Long Context Document-Level Machine Translation](https://mlsanity.com/paper/2306.05183/) (2023) - [Quantifying the Plausibility of Context Reliance in Neural Machine Translation](https://mlsanity.com/paper/2310.01188/) (2023) - [Encoding Sentence Position in Context-Aware Neural Machine Translation with Concatenation](https://mlsanity.com/paper/2302.06459/) (2023) - [Text revision in Scientific Writing Assistance: An Overview](https://mlsanity.com/paper/2303.16726/) (2023) - [Analyzing the Attention Heads for Pronoun Disambiguation in Context-aware Machine Translation Models](https://mlsanity.com/paper/2412.11187/) (2024) - [You Are What You Train: Effects of Data Composition on Training Context-aware Machine Translation Models](https://mlsanity.com/paper/2509.14031/) (2025) - [Sequence Shortening for Context-Aware Machine Translation](https://mlsanity.com/paper/2402.01416/) (2024) - [On Search Strategies for Document-Level Neural Machine Translation](https://mlsanity.com/paper/2306.05116/) (2023) - [Promoting Target Data in Context-aware Neural Machine Translation](https://mlsanity.com/paper/2402.06342/) (2024) - [Scaling Attention Head Analysis via Gradient-Based Attribution in Context-Aware Machine Translation](https://mlsanity.com/paper/2609.28117/) (2026) ## Abstract (arXiv) This paper addresses the task of contextual translation using multi-segment models. Specifically we show that increasing model capacity further pushes the limits of this approach and that deeper models are more suited to capture context dependencies. Furthermore, improvements observed with larger models can be transferred to smaller models using knowledge distillation. Our experiments show that this approach achieves competitive performance across several languages and benchmarks, without additional language-specific tuning and task specific architectures.