mlsanityData to 7 Oct 2026

How to Dissect a Muppet: The Structure of Transformer Embedding Spaces

arXiv 2206.03529 ↗First posted 7 Jun 2022cs.CLcited by 19

Not on the map. Why?

Abstract (arXiv)

Pretrained embeddings based on the Transformer architecture have taken the NLP community by storm. We show that they can mathematically be reframed as a sum of vector factors and showcase how to use this reframing to study the impact of each component. We provide evidence that multi-head attentions and feed-forwards are not equally useful in all downstream applications, as well as a quantitative overview of the effects of finetuning on the overall embedding space. This approach allows us to draw connections to a wide range of previous studies, from vector space anisotropy to attention weights.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 25 of its 46 references that are in the map. First posted on arXiv 7 Jun 2022.

Builds on

25 of 46 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2016Layer Normalizationcited by 5,579
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  6. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  7. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  8. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  9. 2018Self-Attention with Relative Position Representationscited by 1,112
  10. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  11. 2019Attention is not Explanationcited by 916
  12. 2020A Primer in BERTology: What we know about how BERT workscited by 807
13 more
  1. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  2. 2019Attention is not not Explanationcited by 629
  3. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  4. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  5. 2019Is Attention Interpretable?cited by 387
  6. 2020Probing Pretrained Language Models for Lexical Semanticscited by 156
  7. 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 150
  8. 2019The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectivescited by 139
  9. 2019Learning to Deceive with Attention-Based Explanationscited by 111
  10. 2021Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factorscited by 106
  11. 2019Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddingscited by 55
  12. 2021Of Non-Linearity and Commutativity in BERTcited by 16
  13. 2019Using BERT for Word Sense Disambiguationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 6; 2024: 3; 2025: 6; 2026: 2.

19 papers.

Sort:

By area:

  1. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  2. 2022Analyzing Transformers in Embedding Spacecited by 131
  3. 2023Jump to Conclusions: Short-Cutting Transformers With Linear Transformationscited by 89
  4. 2022Understanding Transformer Memorization Recall Through Idiomscited by 62
  5. 2023The Curious Case of Hallucinatory (Un)answerability: Finding Truths in the Hidden States of Over-Confident Large Language Modelscited by 54
  6. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
  7. 2023Explaining How Transformers Use Context to Build Predictionscited by 42
  8. 2025Linguistic Interpretability of Transformer-based Language Models: a systematic reviewcited by 7
  9. 2023Token-wise Decomposition of Autoregressive Language Model Hidden States for Analyzing Model Predictionscited by 6
  10. 2024Transformer Normalisation Layers and the Independence of Semantic Subspacescited by 6
  11. 2025Analyzing limits for in-context learningcited by 6
  12. 2025Private Memorization Editing: Turning Memorization into a Defense to Strengthen Data Privacy in Large Language Modelscited by 6
7 more
  1. 2026Beyond Outliers: A Data-Free Layer-wise Mixed-Precision Quantization Approach Driven by Numerical and Structural Dual-Sensitivitycited by 5
  2. 2024Isotropy, Clusters, and Classifierscited by 4
  3. 2023Why bother with geometry? On the relevance of linear decompositions of Transformer embeddingscited by 2
  4. 2025Cognitive Activation and Chaotic Dynamics in Large Language Models: A Quasi-Lyapunov Analysis of Reasoning Mechanismscited by 1
  5. 2025From Embeddings to Equations: Genetic-Programming Surrogates for Interpretable Transformer Classificationcited by 1
  6. 2025Language Models Learn Universal Representations of Numbers and Here's Why You Should Carecited by 1
  7. 2026Structure Retention in Embedding Spaces as a Predictor of Benchmark Performancecited by 0