mlsanityData to 7 Oct 2026

On Identifiability in Transformers

arXiv 1908.04211 ↗First posted 12 Aug 2019cs.CL cs.LGcited by 115

Not on the map. Why?

Abstract (arXiv)

In this paper we delve deep in the Transformer architecture by investigating two of its core components: self-attention and contextual embeddings. In particular, we study the identifiability of attention weights and token embeddings, and the aggregation of context into hidden tokens. We show that, for sequences longer than the attention head dimension, attention weights are not identifiable. We propose effective attention as a complementary tool for improving explanatory interpretations based on attention. Furthermore, we show that input tokens retain to a large degree their identity across the model. We also find evidence suggesting that identity information is mainly encoded in the angle of the embeddings and gradually decreases with depth. Finally, we demonstrate strong mixing of input information in the generation of contextual embeddings by means of a novel quantification method based on gradient attribution. Overall, we show that self-attention distributions are not directly interpretable and present tools to better understand and further investigate Transformer models.

Key facts. Cited by 115 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 31; 2022: 21; 2023: 19; 2024: 16; 2025: 16; 2026: 12.

115 papers.

Sort:

By area:

  1. 2023Explainability for Large Language Models: A Surveycited by 331
  2. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
  3. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 230
  4. 2021Post-hoc Interpretability for Neural NLP: A Surveycited by 155
  5. 2025Why do LLMs attend to the first token?cited by 122
  6. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  7. 2025SAEs Are Good for Steering -- If You Select the Right Featurescited by 74
  8. 2021A Survey on Interpretable Reinforcement Learningcited by 70
  9. 2022Measuring the Mixing of Contextual Information in the Transformercited by 63
  10. 2022Entropy-based Attention Regularization Frees Unintended Bias Mitigation from Listscited by 54
  11. 2021Incorporating Residual and Normalization Layers into Analysis of Masked Language Modelscited by 50
  12. 2023Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Modelscited by 50
38 more
  1. 2022Towards Opening the Black Box of Neural Machine Translation: Source and Target Interpretations of the Transformercited by 45
  2. 2022GlobEnc: Quantifying Global Token Attribution by Incorporating the Whole Encoder Layer in Transformerscited by 44
  3. 2022Rethinking Attention-Model Explainability through Faithfulness Violation Testcited by 43
  4. 2023Quantifying Context Mixing in Transformerscited by 43
  5. 2023Explaining How Transformers Use Context to Build Predictionscited by 42
  6. 2021Approximating How Single Head Attention Learnscited by 38
  7. 2022On the Explainability of Natural Language Processing Deep Modelscited by 37
  8. 2021Order in the Court: Explainable AI Methods Prone to Disagreementcited by 36
  9. 2021Distilling Linguistic Context for Language Model Compressioncited by 33
  10. 2023Analyzing Feed-Forward Blocks in Transformers through the Lens of Attention Mapscited by 33
  11. 2021Is Sparse Attention more Interpretable?cited by 31
  12. 2023DecompX: Explaining Transformers Decisions by Propagating Token Decompositioncited by 31
  13. 2023Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammarscited by 30
  14. 2022Attention Mechanism in Neural Networks: Where it Comes and Where it Goescited by 29
  15. 2021Dodrio: Exploring Transformer Models with Interactive Visualizationcited by 28
  16. 2025Large Language Models Are Human-Like Internallycited by 28
  17. 2021Rationales for Sequential Predictionscited by 26
  18. 2023AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compressioncited by 26
  19. 2023Analyzing Vision Transformers for Image Classification in Class Embedding Spacecited by 26
  20. 2021Towards BERT-based Automatic ICD Coding: Limitations and Opportunitiescited by 25
  21. 2024Mechanistic?cited by 25
  22. 2021Relative Molecule Self-Attention Transformercited by 21
  23. 2021Multimodal Integration of Human-Like Attention in Visual Question Answeringcited by 20
  24. 2021Cross-lingual Visual Pre-training for Multimodal Machine Translationcited by 19
  25. 2021Interpreting Deep Learning Models in Natural Language Processing: A Reviewcited by 19
  26. 2024Token Transformation Matters: Towards Faithful Post-hoc Explanation for Vision Transformercited by 19
  27. 2022A Unified Understanding of Deep NLP Models for Text Classificationcited by 18
  28. 2025A Comprehensive Survey on Self-Interpretable Neural Networkscited by 18
  29. 2021Of Non-Linearity and Commutativity in BERTcited by 16
  30. 2021Which transformer architecture fits my data? A vocabulary bottleneck in self-attentioncited by 16
  31. 2021Enjoy the Salience: Towards Better Transformer-based Faithful Explanations with Word Saliencecited by 16
  32. 2023COCKATIEL: COntinuous Concept ranKed ATtribution with Interpretable ELements for explaining neural net classifiers on NLP taskscited by 16
  33. 2021Explainable Deep Learning in Healthcare: A Methodological Survey from an Attribution Viewcited by 15
  34. 2021On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformerscited by 14
  35. 2022A Song of (Dis)agreement: Evaluating the Evaluation of Explainable Artificial Intelligence in Natural Language Processingcited by 13
  36. 2021Effective Attention Sheds Light On Interpretabilitycited by 11
  37. 2021Attention Weights in Transformer NMT Fail Aligning Words Between Sequences but Largely Explain Model Predictionscited by 11
  38. 2022eX-ViT: A Novel eXplainable Vision Transformer for Weakly Supervised Semantic Segmentationcited by 11

50 most cited of 115 All 115 (JSON)