mlsanityData to 7 Oct 2026

Analyzing Transformers in Embedding Space

arXiv 2209.02535 ↗First posted 6 Sep 2022cs.CL cs.LGcited by 131

Not on the map. Why?

Abstract (arXiv)

Understanding Transformer-based models has attracted significant attention, as they lie at the heart of recent technological advances across machine learning. While most interpretability methods rely on running models over inputs, recent work has shown that a zero-pass approach, where parameters are interpreted directly without a forward/backward pass is feasible for some Transformer parameters, and for two-layer attention networks. In this work, we present a theoretical analysis where all parameters of a trained Transformer are interpreted by projecting them into the embedding space, that is, the space of vocabulary items they operate on. We derive a simple theoretical framework to support our arguments and provide ample evidence for its validity. First, an empirical analysis showing that parameters of both pretrained and fine-tuned models can be interpreted in embedding space. Second, we present two applications of our framework: (a) aligning the parameters of different models that share a vocabulary, and (b) constructing a classifier without training by ``translating'' the parameters of a fine-tuned classifier to parameters of a different model that was only pretrained. Overall, our findings open the door to interpretation methods that, at least in part, abstract away from model specifics and operate in the embedding space only.

Key facts. Cited by 131 papers in the map (data through 7 Oct 2026). Builds on 28 of its 60 references that are in the map. First posted on arXiv 6 Sep 2022.

Builds on

28 of 60 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  9. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  10. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  11. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  12. 2020A Primer in BERTology: What we know about how BERT workscited by 807
16 more
  1. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  2. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  3. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  4. 2016Using the Output Embedding to Improve Language Modelscited by 266
  5. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 250
  6. 2014Understanding image representations by measuring their equivariance and equivalencecited by 240
  7. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  8. 2021Revisiting Model Stitching to Compare Neural Representationscited by 203
  9. 2019The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectivescited by 139
  10. 2018What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Modelscited by 124
  11. 2020Analyzing Individual Neurons in Pre-trained Language Modelscited by 97
  12. 2019Augmenting Self-attention with Persistent Memorycited by 92
  13. 2021The MultiBERTs: BERT Reproductions for Robustness Analysiscited by 89
  14. 2021Similarity and Matching of Neural Network Representationscited by 77
  15. 2022LM-Debugger: An Interactive Tool for Inspection and Intervention in Transformer-Based Language Modelscited by 28
  16. 2022How to Dissect a Muppet: The Structure of Transformer Embedding Spacescited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 26; 2024: 40; 2025: 40; 2026: 22.

131 papers.

Sort:

By area:

  1. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  2. 2022Mass-Editing Memory in a Transformercited by 976
  3. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  4. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  5. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  6. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  7. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  8. 2023Function Vectors in Large Language Modelscited by 292
  9. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 225
  10. 2026LTX-2: Efficient Joint Audio-Visual Foundation Modelcited by 213
  11. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  12. 2023Birth of a Transformer: A Memory Viewpointcited by 154
38 more
  1. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 145
  2. 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 145
  3. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  4. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  5. 2024Playground v3: Improving Text-to-Image Alignment with Deep-Fusion Large Language Modelscited by 113
  6. 2024Universal Neurons in GPT2 Language Modelscited by 100
  7. 2024Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristicscited by 94
  8. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  9. 2023Jump to Conclusions: Short-Cutting Transformers With Linear Transformationscited by 89
  10. 2023Copy Suppression: Comprehensively Understanding an Attention Headcited by 73
  11. 2023Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Modelscited by 67
  12. 2024Cross-modal Information Flow in Multimodal Large Language Modelscited by 65
  13. 2024Confidence Regulation Neurons in Language Modelscited by 57
  14. 2023From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuningcited by 56
  15. 2024Spectral Filters, Dark Signals, and Attention Sinkscited by 54
  16. 2024From Understanding to Utilization: A Survey on Explainability for Large Language Modelscited by 47
  17. 2024From Tokens to Words: On the Inner Lexicon of LLMscited by 46
  18. 2023Self-Detoxifying Language Models via Toxification Reversalcited by 45
  19. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
  20. 2024In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigationcited by 43
  21. 2023Explaining How Transformers Use Context to Build Predictionscited by 42
  22. 2024Locate-then-edit for Multi-hop Factual Recall under Knowledge Editingcited by 36
  23. 2024Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice Questionscited by 35
  24. 2023A Survey on Knowledge Editing of Neural Networkscited by 34
  25. 2024HiVG: Hierarchical Multimodal Fine-grained Modulation for Visual Groundingcited by 31
  26. 2024UniBias: Unveiling and Mitigating LLM Bias through Internal Attention and FFN Manipulationcited by 30
  27. 2022What Are You Token About? Dense Retrieval as Distributions Over the Vocabularycited by 28
  28. 2024Intrinsic Test of Unlearning Using Parametric Knowledge Tracescited by 28
  29. 2025Large Language Models Are Human-Like Internallycited by 28
  30. 2023Analyzing Vision Transformers for Image Classification in Class Embedding Spacecited by 26
  31. 2023Overcoming Generic Knowledge Loss with Selective Parameter Updatecited by 23
  32. 2025Back Attention: Understanding and Enhancing Multi-Hop Reasoning in Large Language Modelscited by 23
  33. 2023Analyzing And Editing Inner Mechanisms Of Backdoored Language Modelscited by 22
  34. 2023DecoderLens: Layerwise Interpretation of Encoder-Decoder Transformerscited by 22
  35. 2023Memory Injections: Correcting Multi-Hop Reasoning Failures during Inference in Transformer-Based Language Modelscited by 21
  36. 2023Attention Lens: A Tool for Mechanistically Interpreting the Attention Head Information Retrieval Mechanismcited by 21
  37. 2024Optimal ablation for interpretabilitycited by 21
  38. 2024An Investigation of Neuron Activation as a Unified Lens to Explain Chain-of-Thought Eliciting Arithmetic Reasoning of LLMscited by 20

50 most cited of 131 All 131 (JSON)