# Encoding Sentence Position in Context-Aware Neural Machine Translation with Concatenation

arXiv: https://arxiv.org/abs/2302.06459 · HTML: https://mlsanity.com/paper/2302.06459/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 42 references that are in the map.
- First posted on arXiv 13 Feb 2023.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [Context-Aware Neural Machine Translation Learns Anaphora Resolution](https://mlsanity.com/paper/1805.10163/) (2018)
- [Neural Machine Translation with Extended Context](https://mlsanity.com/paper/1708.05943/) (2017)
- [Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation](https://mlsanity.com/paper/1808.07048/) (2018)
- [Training Tips for the Transformer Model](https://mlsanity.com/paper/1804.00247/) (2018)
- [Learning to Encode Position for Transformer with Continuous Dynamical Model](https://mlsanity.com/paper/2003.09229/) (2020)
- [Selective Attention for Context-aware Neural Machine Translation](https://mlsanity.com/paper/1903.08788/) (2019)
- [Microsoft Translator at WMT 2019: Towards Large-Scale Document-Level Neural Machine Translation](https://mlsanity.com/paper/1907.06170/) (2019)
- [Measuring and Increasing Context Usage in Context-Aware Machine Translation](https://mlsanity.com/paper/2105.03482/) (2021)
- [G-Transformer for Document-level Machine Translation](https://mlsanity.com/paper/2105.14761/) (2021)
- [When and Why is Document-level Context Useful in Neural Machine Translation?](https://mlsanity.com/paper/1910.00294/) (2019)
- [Rethinking Document-level Neural Machine Translation](https://mlsanity.com/paper/2010.08961/) (2020)
- [Long-Short Term Masking Transformer: A Simple but Effective Baseline for Document-level Neural Machine Translation](https://mlsanity.com/paper/2009.09127/) (2020)
- [Dynamic Context Selection for Document-level Neural Machine Translation via Reinforcement Learning](https://mlsanity.com/paper/2010.04314/) (2020)
- [Diving Deep into Context-Aware Neural Machine Translation](https://mlsanity.com/paper/2010.09482/) (2020)
- [Divide and Rule: Effective Pre-Training for Context-Aware Multi-Encoder Translation Models](https://mlsanity.com/paper/2103.17151/) (2021)
- [A baseline revisited: Pushing the limits of multi-segment models for context-aware translation](https://mlsanity.com/paper/2210.10906/) (2022)
- [A Comparison of Approaches to Document-level Machine Translation](https://mlsanity.com/paper/2101.11040/) (2021)
- [Focused Concatenation for Context-Aware Neural Machine Translation](https://mlsanity.com/paper/2210.13388/) (2022)

## Built on by (most cited first, 6 of 6)

- [Escaping the sentence-level paradigm in machine translation](https://mlsanity.com/paper/2304.12959/) (2023)
- [Findings of the WMT 2023 Shared Task on Discourse-Level Literary Translation: A Fresh Orb in the Cosmos of LLMs](https://mlsanity.com/paper/2311.03127/) (2023)
- [Contextual Refinement of Translations: Large Language Models for Sentence and Document-Level Post-Editing](https://mlsanity.com/paper/2310.14855/) (2023)
- [Investigating Length Issues in Document-level Machine Translation](https://mlsanity.com/paper/2412.17592/) (2024)
- [Improving Large Language Model (LLM) fidelity through context-aware grounding: A systematic approach to reliability and veracity](https://mlsanity.com/paper/2408.04023/) (2024)
- [Promoting Target Data in Context-aware Neural Machine Translation](https://mlsanity.com/paper/2402.06342/) (2024)

## Abstract (arXiv)

Context-aware translation can be achieved by processing a concatenation of consecutive sentences with the standard Transformer architecture. This paper investigates the intuitive idea of providing the model with explicit information about the position of the sentences contained in the concatenation window. We compare various methods to encode sentence positions into token representations, including novel methods. Our results show that the Transformer benefits from certain sentence position encoding methods on English to Russian translation if trained with a context-discounted loss (Lupo et al., 2022). However, the same benefits are not observed in English to German. Further empirical efforts are necessary to define the conditions under which the proposed approach is beneficial.