mlsanityData to 14 Oct 2026

Do Context-Aware Translation Models Pay the Right Attention?

arXiv 2105.06977 ↗First posted 14 May 2021cs.CL cs.AI cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

Context-aware machine translation models are designed to leverage contextual information, but often fail to do so. As a result, they inaccurately disambiguate pronouns and polysemous words that require context for resolution. In this paper, we ask several questions: What contexts do human translators use to resolve ambiguous words? Are models paying large amounts of attention to the same context? What if we explicitly train them to do so? To answer these questions, we introduce SCAT (Supporting Context for Ambiguous Translations), a new English-French dataset comprising supporting context words for 14K translations that professional translators found useful for pronoun disambiguation. Using SCAT, we perform an in-depth analysis of the context used to disambiguate, examining positional and lexical characteristics of the supporting words. Furthermore, we measure the degree of alignment between the model's attention scores and the supporting context from SCAT, and apply a guided attention strategy to encourage agreement between the two.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 31 of its 51 references that are in the map. First posted on arXiv 14 May 2021.

Builds on

31 of 51 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  7. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  8. 2021Word Alignment by Fine-tuning Embeddings on Parallel Corporacited by 186
  9. 2018Context-Aware Neural Machine Translation Learns Anaphora Resolutioncited by 97
  10. 2017Neural Machine Translation with Extended Contextcited by 95
  11. 2020Evaluating Explanations: How much do explanations from the teacher aid students?cited by 94
  12. 2018Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluationcited by 88
19 more
  1. 2017Evaluating Discourse Phenomena in Neural Machine Translationcited by 87
  2. 2018Document-Level Neural Machine Translation with Hierarchical Attention Networkscited by 85
  3. 2018Improving the Transformer Translation Model with Document-Level Contextcited by 74
  4. 2018Linguistically-Informed Self-Attention for Semantic Role Labelingcited by 71
  5. 2019Selective Attention for Context-aware Neural Machine Translationcited by 60
  6. 2018Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translationcited by 58
  7. 2018Deriving Machine Attention from Human Rationalescited by 54
  8. 2017Exploiting Cross-Sentence Context for Neural Machine Translationcited by 48
  9. 2019When and Why is Document-level Context Useful in Neural Machine Translation?cited by 45
  10. 2017Document Context Neural Machine Translation with Memory Networkscited by 41
  11. 2019Context-Aware Monolingual Repair for Neural Machine Translationcited by 41
  12. 2017Learning to Remember Translation History with a Continuous Cachecited by 39
  13. 2017Does Neural Machine Translation Benefit from Larger Context?cited by 38
  14. 2020Does Multi-Encoder Help? A Case Study on Context-Aware Neural Machine Translationcited by 34
  15. 2016Neural Machine Translation with Supervised Attentioncited by 31
  16. 2020Diving Deep into Context-Aware Neural Machine Translationcited by 25
  17. 2019Fine-grained Sentiment Analysis with Faithful Attentioncited by 23
  18. 2020Modeling Discourse Structure for Document-level Neural Machine Translationcited by 11
  19. 2016Supervised Attentions for Neural Machine Translationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 4; 2023: 6; 2024: 5; 2025: 3; 2026: 2.

21 papers.

Sort:

By area:

  1. 2022Interpreting Language Models with Contrastive Explanationscited by 92
  2. 2021When Does Translation Require Context? A Data-driven, Multilingual Explorationcited by 36
  3. 2023A Tale of Pronouns: Interpretability Informs Gender Bias Mitigation for Fairer Instruction-Tuned Machine Translationcited by 19
  4. 2022A baseline revisited: Pushing the limits of multi-segment models for context-aware translationcited by 11
  5. 2023Should We Attend More or Less? Modulating Attention for Fairnesscited by 11
  6. 2024Investigating Length Issues in Document-level Machine Translationcited by 10
  7. 2024On Measuring Context Utilization in Document-Level MT Systemscited by 9
  8. 2023Quantifying the Plausibility of Context Reliance in Neural Machine Translationcited by 7
  9. 2025Are We Paying Attention to Her? Investigating Gender Disambiguation and Attention in Machine Translationcited by 7
  10. 2023XplainLLM: A Knowledge-Augmented Dataset for Reliable Grounded Explanations in LLMscited by 5
  11. 2024Analyzing the Attention Heads for Pronoun Disambiguation in Context-aware Machine Translation Modelscited by 4
  12. 2022Modeling Context With Linear Attention for Scalable Document-Level Translationcited by 3
9 more
  1. 2023That was the last straw, we need more: Are Translation Systems Sensitive to Disambiguating Context?cited by 3
  2. 2025You Are What You Train: Effects of Data Composition on Training Context-aware Machine Translation Modelscited by 3
  3. 2024Sequence Shortening for Context-Aware Machine Translationcited by 2
  4. 2025What Triggers my Model? Contrastive Explanations Inform Gender Choices by Translation Modelscited by 2
  5. 2023Predicting Human Translation Difficulty with Neural Machine Translationcited by 1
  6. 2026Gender Disambiguation in Machine Translation: Diagnostic Evaluation in Decoder-Only Architecturescited by 1
  7. 2026Which Tokens Need Context? A Reference-Based Analysis of Translation Responsibility Using Fertility and Entropycited by 1
  8. 2022Cross-lingual AMR Aligner: Paying Attention to Cross-Attentioncited by 0
  9. 2024Mention Attention for Pronoun Translationcited by 0