mlsanityData to 7 Oct 2026

Explaining How Transformers Use Context to Build Predictions

arXiv 2305.12535 ↗First posted 21 May 2023cs.CL cs.AI cs.LGcited by 42

Not on the map. Why?

Abstract (arXiv)

Language Generation Models produce words based on the previous context. Although existing methods offer input attributions as explanations for a model's prediction, it is still unclear how prior words affect the model's decision throughout the layers. In this work, we leverage recent advances in explainability of the Transformer and present a procedure to analyze models for language generation. Using contrastive examples, we compare the alignment of our explanations with evidence of the linguistic phenomena, and show that our method consistently aligns better than gradient-based and perturbation-based baselines. Then, we investigate the role of MLPs inside the Transformer and show that they learn features that help the model predict words that are grammatically acceptable. Lastly, we apply our method to Neural Machine Translation models, and demonstrate that they generate human-like source-target alignments for building predictions.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 39 of its 52 references that are in the map. First posted on arXiv 21 May 2023.

Builds on

39 of 52 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  5. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  6. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  7. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  8. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  9. 2022In-context Learning and Induction Headscited by 930
  10. 2019Attention is not Explanationcited by 916
  11. 2020Quantifying Attention Flow in Transformerscited by 759
  12. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
27 more
  1. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  2. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  3. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  4. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  5. 2019Is Attention Interpretable?cited by 387
  6. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
  7. 2016Not Just a Black Box: Learning Important Features Through Propagating Activation Differencescited by 295
  8. 2016Understanding Neural Networks through Representation Erasurecited by 223
  9. 2015Visualizing and Understanding Neural Models in NLPcited by 199
  10. 2020A Diagnostic Study of Explainability Techniques for Text Classificationcited by 158
  11. 2019The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectivescited by 139
  12. 2022Analyzing Transformers in Embedding Spacecited by 131
  13. 2019On Identifiability in Transformerscited by 115
  14. 2019Learning to Deceive with Attention-Based Explanationscited by 111
  15. 2022Interpreting Language Models with Contrastive Explanationscited by 91
  16. 2023Jump to Conclusions: Short-Cutting Transformers With Linear Transformationscited by 89
  17. 2023Inseq: An Interpretability Toolkit for Sequence Generation Modelscited by 78
  18. 2019Jointly Learning to Align and Translate with Transformer Modelscited by 75
  19. 2022Measuring the Mixing of Contextual Information in the Transformercited by 63
  20. 2014Extraction of Salient Sentences from Labelled Documentscited by 50
  21. 2021Incorporating Residual and Normalization Layers into Analysis of Masked Language Modelscited by 50
  22. 2022Towards Opening the Black Box of Neural Machine Translation: Source and Target Interpretations of the Transformercited by 45
  23. 2023Quantifying Context Mixing in Transformerscited by 43
  24. 2019Adding Interpretable Attention to Neural Translation Models Improves Word Alignmentcited by 30
  25. 2019Saliency-driven Word Alignment Interpretation for Neural Machine Translationcited by 27
  26. 2022How to Dissect a Muppet: The Structure of Transformer Embedding Spacescited by 19
  27. 2022A Multilingual Perspective Towards the Evaluation of Attribution Methods in Natural Language Inferencecited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 14; 2025: 17; 2026: 5.

42 papers.

Sort:

By area:

  1. 2023Neurons in Large Language Models: Dead, N-gram, Positionalcited by 88
  2. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
  3. 2024On Large Language Models' Hallucination with Regard to Known Factscited by 43
  4. 2024Sharing Matters: Analysing Neurons Across Languages and Tasks in LLMscited by 33
  5. 2024Model Internals-based Answer Attribution for Trustworthy Retrieval-Augmented Generationcited by 25
  6. 2024The Hidden Space of Transformer Language Adapterscited by 16
  7. 2025LLMs as a synthesis between symbolic and distributed approaches to languagecited by 13
  8. 2024Correcting Negative Bias in Large Language Models through Negative Attention Score Alignmentcited by 12
  9. 2025Attributing Response to Context: A Jensen-Shannon Divergence Driven Mechanistic Study of Context Attribution in Retrieval-Augmented Generationcited by 12
  10. 2024Interpreting Context Look-ups in Transformers: Investigating Attention-MLP Interactionscited by 11
  11. 2025A Close Look at Decomposition-based XAI-Methods for Transformer Language Modelscited by 10
  12. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
30 more
  1. 2025InvisibleInk: High-Utility and Low-Cost Text Generation with Differential Privacycited by 10
  2. 2023Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attentioncited by 7
  3. 2023Quantifying the Plausibility of Context Reliance in Neural Machine Translationcited by 6
  4. 2025GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMscited by 6
  5. 2025Decoupling Positional and Symbolic Attention Behavior in Transformerscited by 6
  6. 2024A Large Encoder-Decoder Family of Foundation Models For Chemical Languagecited by 5
  7. 2024Context-Aware or Context-Insensitive? Assessing LLMs' Performance in Document-Level Translationcited by 5
  8. 2025B-cos LM: Efficiently Transforming Pre-trained Language Models for Improved Explainabilitycited by 4
  9. 2025Unsupervised Word-level Quality Estimation for Machine Translation Through the Lens of Annotators (Dis)agreementcited by 4
  10. 2025How Programming Concepts and Neurons Are Shared in Code Language Modelscited by 4
  11. 2024Counterfactuals As a Means for Evaluating Faithfulness of Attribution Methods in Autoregressive Language Modelscited by 3
  12. 2025LaTIM: Measuring Latent Token-to-Token Interactions in Mamba Modelscited by 3
  13. 2025The Unheard Alternative: Contrastive Explanations for Speech-to-Text Modelscited by 3
  14. 2023Why bother with geometry? On the relevance of linear decompositions of Transformer embeddingscited by 2
  15. 2023Explainable Identification of Hate Speech towards Islam using Graph Neural Networkscited by 2
  16. 2024Bridging Linguistic Structure and Mechanistic Interpretability for Conceptual Interpretation in Language Modelscited by 2
  17. 2025Which Words Matter Most in Zero-Shot Prompts?cited by 2
  18. 2025Steering Information Utility in Key-Value Memory for Language Model Post-Trainingcited by 2
  19. 2024When Only Time Will Tell: Interpreting How Transformers Process Local Ambiguities Through the Lens of Restart-Incrementalitycited by 1
  20. 2024Analyzing Context Contributions in LLM-based Machine Translationcited by 1
  21. 2024On Explaining with Attention Matricescited by 1
  22. 2025DePass: Unified Feature Attributing by Simple Decomposed Forward Passcited by 1
  23. 2026Contrastive Attribution in the Wild: An Interpretability Analysis of LLM Failures on Realistic Benchmarkscited by 1
  24. 2023Let the Models Respond: Interpreting Language Model Detoxification Through the Lens of Prompt Dependencecited by 0
  25. 2025Beyond the Sentence: A Survey on Context-Aware Machine Translation with Large Language Modelscited by 0
  26. 2025Mapping Scientific Literature with Large Language Models and Topic Modelingcited by 0
  27. 2026LANCET: Neural Intervention via Structural Entropy for Mitigating Faithfulness Hallucinations in LLMscited by 0
  28. 2026Internal Flow Signatures for Self-Checking and Refinement in LLMscited by 0
  29. 2026Layer-wise Target Propagation: Efficient Component Attribution through Target Centric Propagationcited by 0
  30. 2026Grad Detect: Gradient-Based Hallucination Detection in LLMscited by 0