# G-Transformer for Document-level Machine Translation

arXiv: https://arxiv.org/abs/2105.14761 · HTML: https://mlsanity.com/paper/2105.14761/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 47 references that are in the map.
- First posted on arXiv 31 May 2021.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [Six Challenges for Neural Machine Translation](https://mlsanity.com/paper/1706.03872/) (2017)
- [Neural Machine Translation with Extended Context](https://mlsanity.com/paper/1708.05943/) (2017)
- [Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation](https://mlsanity.com/paper/1808.07048/) (2018)
- [Document-Level Neural Machine Translation with Hierarchical Attention Networks](https://mlsanity.com/paper/1809.01576/) (2018)
- [Improving the Transformer Translation Model with Document-Level Context](https://mlsanity.com/paper/1810.03581/) (2018)
- [Selective Attention for Context-aware Neural Machine Translation](https://mlsanity.com/paper/1903.08788/) (2019)
- [Microsoft Translator at WMT 2019: Towards Large-Scale Document-Level Neural Machine Translation](https://mlsanity.com/paper/1907.06170/) (2019)
- [Document Context Neural Machine Translation with Memory Networks](https://mlsanity.com/paper/1711.03688/) (2017)
- [Context-Aware Monolingual Repair for Neural Machine Translation](https://mlsanity.com/paper/1909.01383/) (2019)
- [Long-Short Term Masking Transformer: A Simple but Effective Baseline for Document-level Neural Machine Translation](https://mlsanity.com/paper/2009.09127/) (2020)
- [Overcoming the Curse of Sentence Length for Neural Machine Translation using Automatic Segmentation](https://mlsanity.com/paper/1409.1257/) (2014)
- [Context Gates for Neural Machine Translation](https://mlsanity.com/paper/1608.06043/) (2016)
- [Enhancing Context Modeling with a Query-Guided Capsule Network for Document-level Translation](https://mlsanity.com/paper/1909.00564/) (2019)
- [Contextualized Rewriting for Text Summarization](https://mlsanity.com/paper/2102.00385/) (2021)

## Built on by (most cited first, 30 of 46)

- [Document-Level Machine Translation with Large Language Models](https://mlsanity.com/paper/2304.02210/) (2023)
- [Cross-modal Prototype Driven Network for Radiology Report Generation](https://mlsanity.com/paper/2207.04818/) (2022)
- [Escaping the sentence-level paradigm in machine translation](https://mlsanity.com/paper/2304.12959/) (2023)
- [DelTA: An Online Document-Level Translation Agent Based on Multi-Level Memory](https://mlsanity.com/paper/2410.08143/) (2024)
- [Efficiently Exploring Large Language Models for Document-Level Machine Translation with In-context Learning](https://mlsanity.com/paper/2406.07081/) (2024)
- [Contextual Refinement of Translations: Large Language Models for Sentence and Document-Level Post-Editing](https://mlsanity.com/paper/2310.14855/) (2023)
- [Learn To Remember: Transformer with Recurrent Memory for Document-Level Machine Translation](https://mlsanity.com/paper/2205.01546/) (2022)
- [Document Flattening: Beyond Concatenating Context for Document-Level Neural Machine Translation](https://mlsanity.com/paper/2302.08079/) (2023)
- [Identifying Context-Dependent Translations for Evaluation Set Production](https://mlsanity.com/paper/2311.02321/) (2023)
- [Enhancing Document-level Translation of Large Language Model via Translation Mixed-instructions](https://mlsanity.com/paper/2401.08088/) (2024)
- [A baseline revisited: Pushing the limits of multi-segment models for context-aware translation](https://mlsanity.com/paper/2210.10906/) (2022)
- [Investigating Length Issues in Document-level Machine Translation](https://mlsanity.com/paper/2412.17592/) (2024)
- [Improving Long Context Document-Level Machine Translation](https://mlsanity.com/paper/2306.05183/) (2023)
- [Focused Concatenation for Context-Aware Neural Machine Translation](https://mlsanity.com/paper/2210.13388/) (2022)
- [Encoding Sentence Position in Context-Aware Neural Machine Translation with Concatenation](https://mlsanity.com/paper/2302.06459/) (2023)
- [DeMPT: Decoding-enhanced Multi-phase Prompt Tuning for Making LLMs Be Better Context-aware Translators](https://mlsanity.com/paper/2402.15200/) (2024)
- [Sketching as a Tool for Understanding and Accelerating Self-attention for Long Sequences](https://mlsanity.com/paper/2112.05359/) (2021)
- [P-Transformer: Towards Better Document-to-Document Neural Machine Translation](https://mlsanity.com/paper/2212.05830/) (2022)
- [Target-Side Augmentation for Document-Level Machine Translation](https://mlsanity.com/paper/2305.04505/) (2023)
- [Core Building Blocks: Next Gen Geo Spatial GPT Application](https://mlsanity.com/paper/2310.11029/) (2023)
- [Importance-Aware Data Augmentation for Document-Level Neural Machine Translation](https://mlsanity.com/paper/2401.15360/) (2024)
- [A Case Study on Context-Aware Neural Machine Translation with Multi-Task Learning](https://mlsanity.com/paper/2407.03076/) (2024)
- [Analyzing the Attention Heads for Pronoun Disambiguation in Context-aware Machine Translation Models](https://mlsanity.com/paper/2412.11187/) (2024)
- [You Are What You Train: Effects of Data Composition on Training Context-aware Machine Translation Models](https://mlsanity.com/paper/2509.14031/) (2025)
- [Advancing Multilingual Pre-training: TRIP Triangular Document-level Pre-training for Multilingual Language Models](https://mlsanity.com/paper/2212.07752/) (2022)
- [MUTANT: A Multi-sentential Code-mixed Hinglish Dataset](https://mlsanity.com/paper/2302.11766/) (2023)
- [Exploring Paracrawl for Document-level Neural Machine Translation](https://mlsanity.com/paper/2304.10216/) (2023)
- [Sequence Shortening for Context-Aware Machine Translation](https://mlsanity.com/paper/2402.01416/) (2024)
- [Context-Aware Machine Translation with Source Coreference Explanation](https://mlsanity.com/paper/2404.19505/) (2024)
- [Improving LLM-based Document-level Machine Translation with Multi-Knowledge Fusion](https://mlsanity.com/paper/2503.12152/) (2025)

## Abstract (arXiv)

Document-level MT models are still far from satisfactory. Existing work extend translation unit from single sentence to multiple sentences. However, study shows that when we further enlarge the translation unit to a whole document, supervised training of Transformer can fail. In this paper, we find such failure is not caused by overfitting, but by sticking around local minima during training. Our analysis shows that the increased complexity of target-to-source attention is a reason for the failure. As a solution, we propose G-Transformer, introducing locality assumption as an inductive bias into Transformer, reducing the hypothesis space of the attention from target to source. Experiments show that G-Transformer converges faster and more stably than Transformer, achieving new state-of-the-art BLEU scores for both non-pretraining and pre-training settings on three benchmark datasets.