mlsanityData to 14 Oct 2026

Towards Opening the Black Box of Neural Machine Translation: Source and Target Interpretations of the Transformer

arXiv 2205.11631 ↗First posted 23 May 2022cs.CLcited by 45

Not on the map. Why?

Abstract (arXiv)

In Neural Machine Translation (NMT), each token prediction is conditioned on the source sentence and the target prefix (what has been previously translated at a decoding step). However, previous work on interpretability in NMT has mainly focused solely on source sentence tokens' attributions. Therefore, we lack a full understanding of the influences of every input token (source sentence and target prefix) in the model predictions. In this work, we propose an interpretability method that tracks input tokens' attributions for both contexts. Our method, which can be extended to any encoder-decoder Transformer-based model, allows us to better comprehend the inner workings of current NMT models. We apply the proposed method to both bilingual and multilingual Transformers and present insights into their behaviour.

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). Builds on 28 of its 37 references that are in the map. First posted on arXiv 23 May 2022.

Builds on

28 of 37 references.

References · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  8. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  9. 2019Attention is not Explanationcited by 929
  10. 2020Quantifying Attention Flow in Transformerscited by 775
  11. 2020Transformer Interpretability Beyond Attention Visualizationcited by 627
  12. 2020Beyond English-Centric Multilingual Machine Translationcited by 561
16 more
  1. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  2. 2019Is Attention Interpretable?cited by 389
  3. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 320
  4. 2019The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectivescited by 139
  5. 2019On Identifiability in Transformerscited by 116
  6. 2018Context-Aware Neural Machine Translation Learns Anaphora Resolutioncited by 97
  7. 2019Jointly Learning to Align and Translate with Transformer Modelscited by 75
  8. 2020Analyzing the Source and Target Contributions to Predictions in Neural Machine Translationcited by 71
  9. 2019Domain Robustness in Neural Machine Translationcited by 65
  10. 2022Measuring the Mixing of Contextual Information in the Transformercited by 65
  11. 2021Incorporating Residual and Normalization Layers into Analysis of Masked Language Modelscited by 51
  12. 2020Accurate Word Alignment Induction from Neural Machine Translationcited by 32
  13. 2019Adding Interpretable Attention to Neural Translation Models Improves Word Alignmentcited by 30
  14. 2019Saliency-driven Word Alignment Interpretation for Neural Machine Translationcited by 27
  15. 2021Language Modeling, Lexical Translation, Reordering: The Training Process of NMT through the Lens of Classical SMTcited by 21
  16. 2021Attention Weights in Transformer NMT Fail Aligning Words Between Sequences but Largely Explain Model Predictionscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 17; 2024: 10; 2025: 7; 2026: 7.

45 papers.

Sort:

By area:

  1. 2023Hallucinations in Large Multilingual Translation Modelscited by 129
  2. 2023Inseq: An Interpretability Toolkit for Sequence Generation Modelscited by 78
  3. 2022Detecting and Mitigating Hallucinations in Machine Translation: Model Internal Workings Alone Do Well, Sentence Similarity Even Bettercited by 75
  4. 2023On Early Detection of Hallucinations in Factual Question Answeringcited by 56
  5. 2023Explaining How Transformers Use Context to Build Predictionscited by 43
  6. 2023HalOmi: A Manually Annotated Benchmark for Multilingual Hallucination and Omission Detection in Machine Translationcited by 35
  7. 2022Attention as a Guide for Simultaneous Speech Translationcited by 32
  8. 2023Improving Translation Faithfulness of Large Language Models via Augmenting Instructionscited by 28
  9. 2022Toxicity in Multilingual Machine Translation at Scalecited by 26
  10. 2023Multilingual Holistic Bias: Extending Descriptors and Patterns to Unveil Demographic Biases in Languages at Scalecited by 24
  11. 2023DecoderLens: Layerwise Interpretation of Encoder-Decoder Transformerscited by 22
  12. 2023A Tale of Pronouns: Interpretability Informs Gender Bias Mitigation for Fairer Instruction-Tuned Machine Translationcited by 19
33 more
  1. 2024An Audit on the Perspectives and Challenges of Hallucinations in NLPcited by 17
  2. 2022Optimal Transport for Unsupervised Hallucination Detection in Neural Machine Translationcited by 13
  3. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  4. 2024On Measuring Context Utilization in Document-Level MT Systemscited by 9
  5. 2024Enhanced Hallucination Detection in Neural Machine Translation through Simple Detector Aggregationcited by 9
  6. 2024Detecting Hallucination and Coverage Errors in Retrieval Augmented Generation for Controversial Topicscited by 9
  7. 2023Quantifying the Plausibility of Context Reliance in Neural Machine Translationcited by 7
  8. 2023Perturbation-based QE: An Explainable, Unsupervised Word-level Quality Estimation Method for Blackbox Machine Translationcited by 6
  9. 2025Mitigating Hallucinated Translations in Large Language Models with Hallucination-focused Preference Optimizationcited by 6
  10. 2025Hallucinated Span Detection with Multi-View Attention Featurescited by 6
  11. 2023Added Toxicity Mitigation at Inference Time for Multimodal and Massively Multilingual Translationcited by 5
  12. 2025Two Intermediate Translations Are Better Than One: Fine-tuning LLMs for Document-level Translation Refinementcited by 5
  13. 2024Isotropy, Clusters, and Classifierscited by 4
  14. 2024Analyzing the Attention Heads for Pronoun Disambiguation in Context-aware Machine Translation Modelscited by 4
  15. 2026Jacobian Scopes: token-level causal attributions in LLMscited by 4
  16. 2023ReSeTOX: Re-learning attention weights for toxicity mitigation in machine translationcited by 3
  17. 2024Counterfactuals As a Means for Evaluating Faithfulness of Attribution Methods in Autoregressive Language Modelscited by 3
  18. 2025The Unheard Alternative: Contrastive Explanations for Speech-to-Text Modelscited by 3
  19. 2023Computational modeling of semantic changecited by 2
  20. 2023SpeechAlign: a Framework for Speech Translation Alignment Evaluationcited by 2
  21. 2023Why bother with geometry? On the relevance of linear decompositions of Transformer embeddingscited by 2
  22. 2024Unveiling the Role of Pretraining in Direct Speech Translationcited by 2
  23. 2026Cross-Preference Learning for Sentence-Level and Context-Aware Machine Translationcited by 2
  24. 2024What Have We Achieved on Non-autoregressive Translation?cited by 1
  25. 2024Analyzing Context Contributions in LLM-based Machine Translationcited by 1
  26. 2025Cross-Attention is Half Explanation in Speech-to-Text Modelscited by 1
  27. 2026DEX-AR: A Dynamic Explainability Method for Autoregressive Vision-Language Modelscited by 1
  28. 2023Let the Models Respond: Interpreting Language Model Detoxification Through the Lens of Prompt Dependencecited by 0
  29. 2025MMD-Flagger: Leveraging Maximum Mean Discrepancy to Detect Hallucinationscited by 0
  30. 2026Evaluating Explainable AI Attribution Methods in Neural Machine Translation via Attention-Guided Knowledge Distillationcited by 0
  31. 2026Fabricator or dynamic translator?cited by 0
  32. 2026Translation as a Decision Space: A Multi-Agent Perspective on Low-Resource Dialect Generationcited by 0
  33. 2026Scaling Attention Head Analysis via Gradient-Based Attribution in Context-Aware Machine Translationcited by 0