mlsanityData to 14 Oct 2026

Analyzing the Attention Heads for Pronoun Disambiguation in Context-aware Machine Translation Models

arXiv 2412.11187 ↗First posted 15 Dec 2024cs.CL cs.AI cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

In this paper, we investigate the role of attention heads in Context-aware Machine Translation models for pronoun disambiguation in the English-to-German and English-to-French language directions. We analyze their influence by both observing and modifying the attention scores corresponding to the plausible relations that could impact a pronoun prediction. Our findings reveal that while some heads do attend the relations of interest, not all of them influence the models' ability to disambiguate pronouns. We show that certain heads are underutilized by the models, suggesting that model performance could be improved if only the heads would attend one of the relations more strongly. Furthermore, we fine-tune the most promising heads and observe the increase in pronoun disambiguation accuracy of up to 5 percentage points which demonstrates that the improvements in performance can be solidified into the models' parameters.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 42 of its 62 references that are in the map. First posted on arXiv 15 Dec 2024.

Builds on

42 of 62 references.

References · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  3. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  4. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  5. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  6. 2019Attention is not Explanationcited by 929
  7. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  8. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  9. 2020Quantifying Attention Flow in Transformerscited by 775
  10. 2019Attention is not not Explanationcited by 636
  11. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 243
  12. 2021Post-hoc Interpretability for Neural NLP: A Surveycited by 156
30 more
  1. 2019Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)cited by 128
  2. 2019On Identifiability in Transformerscited by 116
  3. 2018Identifying and Controlling Important Neurons in Neural Machine Translationcited by 111
  4. 2017Neural Machine Translation with Extended Contextcited by 95
  5. 2017Evaluating Discourse Phenomena in Neural Machine Translationcited by 87
  6. 2018Document-Level Neural Machine Translation with Hierarchical Attention Networkscited by 85
  7. 2021Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translationcited by 74
  8. 2020Analyzing the Source and Target Contributions to Predictions in Neural Machine Translationcited by 71
  9. 2019Selective Attention for Context-aware Neural Machine Translationcited by 60
  10. 2022Democratizing Neural Machine Translation with OPUS-MTcited by 58
  11. 2021Incorporating Residual and Normalization Layers into Analysis of Masked Language Modelscited by 51
  12. 2021Measuring and Increasing Context Usage in Context-Aware Machine Translationcited by 48
  13. 2021G-Transformer for Document-level Machine Translationcited by 46
  14. 2022Towards Opening the Black Box of Neural Machine Translation: Source and Target Interpretations of the Transformercited by 45
  15. 2023Quantifying Context Mixing in Transformerscited by 43
  16. 2019Context-Aware Monolingual Repair for Neural Machine Translationcited by 41
  17. 2017Does Neural Machine Translation Benefit from Larger Context?cited by 38
  18. 2021When Does Translation Require Context? A Data-driven, Multilingual Explorationcited by 36
  19. 2020Long-Short Term Masking Transformer: A Simple but Effective Baseline for Document-level Neural Machine Translationcited by 27
  20. 2023Escaping the sentence-level paradigm in machine translationcited by 27
  21. 2020Diving Deep into Context-Aware Neural Machine Translationcited by 25
  22. 2021Do Context-Aware Translation Models Pay the Right Attention?cited by 21
  23. 2022Learn To Remember: Transformer with Recurrent Memory for Document-Level Machine Translationcited by 16
  24. 2023Identifying Context-Dependent Translations for Evaluation Set Productioncited by 14
  25. 2022A baseline revisited: Pushing the limits of multi-segment models for context-aware translationcited by 11
  26. 2016Context Gates for Neural Machine Translationcited by 10
  27. 2023Quantifying the Plausibility of Context Reliance in Neural Machine Translationcited by 7
  28. 2020Not all parameters are born equal: Attention is mostly what you needcited by 6
  29. 2021A Dynamic Head Importance Computation Mechanism for Neural Machine Translationcited by 4
  30. 2024Sequence Shortening for Context-Aware Machine Translationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025You Are What You Train: Effects of Data Composition on Training Context-aware Machine Translation Modelscited by 3
  2. 2026Which Tokens Need Context? A Reference-Based Analysis of Translation Responsibility Using Fertility and Entropycited by 1
  3. 2026STAR : Sentence Translation Alignment Rate for Document-to-Document Machine Translationcited by 1
  4. 2026Scaling Attention Head Analysis via Gradient-Based Attribution in Context-Aware Machine Translationcited by 0