# Do Context-Aware Translation Models Pay the Right Attention?

arXiv: https://arxiv.org/abs/2105.06977 · HTML: https://mlsanity.com/paper/2105.06977/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 51 references that are in the map.
- First posted on arXiv 14 May 2021.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Word Alignment by Fine-tuning Embeddings on Parallel Corpora](https://mlsanity.com/paper/2101.08231/) (2021)
- [Context-Aware Neural Machine Translation Learns Anaphora Resolution](https://mlsanity.com/paper/1805.10163/) (2018)
- [Neural Machine Translation with Extended Context](https://mlsanity.com/paper/1708.05943/) (2017)
- [Evaluating Explanations: How much do explanations from the teacher aid students?](https://mlsanity.com/paper/2012.00893/) (2020)
- [Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation](https://mlsanity.com/paper/1808.07048/) (2018)
- [Evaluating Discourse Phenomena in Neural Machine Translation](https://mlsanity.com/paper/1711.00513/) (2017)
- [Document-Level Neural Machine Translation with Hierarchical Attention Networks](https://mlsanity.com/paper/1809.01576/) (2018)
- [Improving the Transformer Translation Model with Document-Level Context](https://mlsanity.com/paper/1810.03581/) (2018)
- [Linguistically-Informed Self-Attention for Semantic Role Labeling](https://mlsanity.com/paper/1804.08199/) (2018)
- [Selective Attention for Context-aware Neural Machine Translation](https://mlsanity.com/paper/1903.08788/) (2019)
- [Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation](https://mlsanity.com/paper/1808.10432/) (2018)
- [Deriving Machine Attention from Human Rationales](https://mlsanity.com/paper/1808.09367/) (2018)
- [Exploiting Cross-Sentence Context for Neural Machine Translation](https://mlsanity.com/paper/1704.04347/) (2017)
- [When and Why is Document-level Context Useful in Neural Machine Translation?](https://mlsanity.com/paper/1910.00294/) (2019)
- [Document Context Neural Machine Translation with Memory Networks](https://mlsanity.com/paper/1711.03688/) (2017)
- [Context-Aware Monolingual Repair for Neural Machine Translation](https://mlsanity.com/paper/1909.01383/) (2019)
- [Learning to Remember Translation History with a Continuous Cache](https://mlsanity.com/paper/1711.09367/) (2017)
- [Does Neural Machine Translation Benefit from Larger Context?](https://mlsanity.com/paper/1704.05135/) (2017)
- [Does Multi-Encoder Help? A Case Study on Context-Aware Neural Machine Translation](https://mlsanity.com/paper/2005.03393/) (2020)
- [Neural Machine Translation with Supervised Attention](https://mlsanity.com/paper/1609.04186/) (2016)
- [Diving Deep into Context-Aware Neural Machine Translation](https://mlsanity.com/paper/2010.09482/) (2020)
- [Fine-grained Sentiment Analysis with Faithful Attention](https://mlsanity.com/paper/1908.06870/) (2019)
- [Modeling Discourse Structure for Document-level Neural Machine Translation](https://mlsanity.com/paper/2006.04721/) (2020)

## Built on by (most cited first, 21 of 21)

- [Interpreting Language Models with Contrastive Explanations](https://mlsanity.com/paper/2202.10419/) (2022)
- [When Does Translation Require Context? A Data-driven, Multilingual Exploration](https://mlsanity.com/paper/2109.07446/) (2021)
- [A Tale of Pronouns: Interpretability Informs Gender Bias Mitigation for Fairer Instruction-Tuned Machine Translation](https://mlsanity.com/paper/2310.12127/) (2023)
- [A baseline revisited: Pushing the limits of multi-segment models for context-aware translation](https://mlsanity.com/paper/2210.10906/) (2022)
- [Should We Attend More or Less? Modulating Attention for Fairness](https://mlsanity.com/paper/2305.13088/) (2023)
- [Investigating Length Issues in Document-level Machine Translation](https://mlsanity.com/paper/2412.17592/) (2024)
- [On Measuring Context Utilization in Document-Level MT Systems](https://mlsanity.com/paper/2402.01404/) (2024)
- [Quantifying the Plausibility of Context Reliance in Neural Machine Translation](https://mlsanity.com/paper/2310.01188/) (2023)
- [Are We Paying Attention to Her? Investigating Gender Disambiguation and Attention in Machine Translation](https://mlsanity.com/paper/2505.08546/) (2025)
- [XplainLLM: A Knowledge-Augmented Dataset for Reliable Grounded Explanations in LLMs](https://mlsanity.com/paper/2311.08614/) (2023)
- [Analyzing the Attention Heads for Pronoun Disambiguation in Context-aware Machine Translation Models](https://mlsanity.com/paper/2412.11187/) (2024)
- [Modeling Context With Linear Attention for Scalable Document-Level Translation](https://mlsanity.com/paper/2210.08431/) (2022)
- [That was the last straw, we need more: Are Translation Systems Sensitive to Disambiguating Context?](https://mlsanity.com/paper/2310.14610/) (2023)
- [You Are What You Train: Effects of Data Composition on Training Context-aware Machine Translation Models](https://mlsanity.com/paper/2509.14031/) (2025)
- [Sequence Shortening for Context-Aware Machine Translation](https://mlsanity.com/paper/2402.01416/) (2024)
- [What Triggers my Model? Contrastive Explanations Inform Gender Choices by Translation Models](https://mlsanity.com/paper/2512.08440/) (2025)
- [Predicting Human Translation Difficulty with Neural Machine Translation](https://mlsanity.com/paper/2312.11852/) (2023)
- [Gender Disambiguation in Machine Translation: Diagnostic Evaluation in Decoder-Only Architectures](https://mlsanity.com/paper/2603.17952/) (2026)
- [Which Tokens Need Context? A Reference-Based Analysis of Translation Responsibility Using Fertility and Entropy](https://mlsanity.com/paper/2606.29489/) (2026)
- [Cross-lingual AMR Aligner: Paying Attention to Cross-Attention](https://mlsanity.com/paper/2206.07587/) (2022)
- [Mention Attention for Pronoun Translation](https://mlsanity.com/paper/2412.14829/) (2024)

## Abstract (arXiv)

Context-aware machine translation models are designed to leverage contextual information, but often fail to do so. As a result, they inaccurately disambiguate pronouns and polysemous words that require context for resolution. In this paper, we ask several questions: What contexts do human translators use to resolve ambiguous words? Are models paying large amounts of attention to the same context? What if we explicitly train them to do so? To answer these questions, we introduce SCAT (Supporting Context for Ambiguous Translations), a new English-French dataset comprising supporting context words for 14K translations that professional translators found useful for pronoun disambiguation. Using SCAT, we perform an in-depth analysis of the context used to disambiguate, examining positional and lexical characteristics of the supporting words. Furthermore, we measure the degree of alignment between the model's attention scores and the supporting context from SCAT, and apply a guided attention strategy to encourage agreement between the two.