mlsanityData to 7 Oct 2026

Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors

arXiv 2103.15949 ↗First posted 29 Mar 2021cs.CL cs.LGcited by 106

Not on the map. Why?

Abstract (arXiv)

Transformer networks have revolutionized NLP representation learning since they were introduced. Though a great effort has been made to explain the representation in transformers, it is widely recognized that our understanding is not sufficient. One important reason is that there lack enough visualization tools for detailed analysis. In this paper, we propose to use dictionary learning to open up these "black boxes" as linear superpositions of transformer factors. Through visualization, we demonstrate the hierarchical semantic structures captured by the transformer factors, e.g., word-level polysemy disambiguation, sentence-level pattern formation, and long-range dependency. While some of these patterns confirm the conventional prior linguistic knowledge, the rest are relatively unexpected, which may provide new insights. We hope this visualization tool can bring further knowledge and a better understanding of how transformer networks work. The code is available at https://github.com/zeyuyun1/TransformerVis

Key facts. Cited by 106 papers in the map (data through 7 Oct 2026). Builds on 18 of its 25 references that are in the map. First posted on arXiv 29 Mar 2021.

Builds on

18 of 25 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  6. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  7. 2019How Can We Know What Language Models Know?cited by 953
  8. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  9. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  10. 2017Dynamic Routing Between Capsulescited by 689
  11. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
  12. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
6 more
  1. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  2. 2019Visualizing and Measuring the Geometry of BERTcited by 215
  3. 2016Linear Algebraic Structure of Word Senses, with Applications to Polysemycited by 165
  4. 2021How to represent part-whole hierarchies in a neural networkcited by 123
  5. 2015Sparse Overcomplete Word Vector Representationscited by 43
  6. 2019Word Embedding Visualization Via Dictionary Learningcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 6; 2023: 7; 2024: 28; 2025: 47; 2026: 14.

106 papers.

Sort:

By area:

  1. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  2. 2024Scaling and evaluating sparse autoencoderscited by 530
  3. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  4. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  5. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 186
  6. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
  7. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  8. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probingcited by 109
  9. 2024Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Controlcited by 83
  10. 2024Sparse autoencoders reveal selective remapping of visual concepts during adaptationcited by 66
  11. 2024Identifying Functionally Important Features with End-to-End Sparse Dictionary Learningcited by 64
  12. 2024InterPLM: Discovering Interpretable Features in Protein Language Models via Sparse Autoencoderscited by 64
38 more
  1. 2023Explaining black box text modules in natural language with language modelscited by 61
  2. 2024Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineeringcited by 60
  3. 2024Confidence Regulation Neurons in Language Modelscited by 57
  4. 2024The Geometry of Concepts: Sparse Autoencoder Feature Structurecited by 52
  5. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  6. 2023Codebook Features: Sparse and Discrete Interpretability for Neural Networkscited by 45
  7. 2024The Missing Curve Detectors of InceptionV1: Applying Sparse Autoencoders to InceptionV1 Early Visioncited by 37
  8. 2025Understanding the Repeat Curse in Large Language Models from a Feature Perspectivecited by 36
  9. 2025Reasoning about Uncertainty: Do Reasoning Models Know When They Don't Know?cited by 35
  10. 2025A Survey on Mechanistic Interpretability for Multi-Modal Foundation Modelscited by 34
  11. 2025The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?cited by 34
  12. 2025LLM Pretraining with Continuous Conceptscited by 33
  13. 2025Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuningcited by 31
  14. 2024One-Step is Enough: Sparse Autoencoders for Text-to-Image Diffusion Modelscited by 30
  15. 2024Out-of-distribution generalization via composition: a lens through induction heads in Transformerscited by 28
  16. 2024Dictionary Learning Improves Patch-Free Circuit Discovery in Mechanistic Interpretability: A Case Study on Othello-GPTcited by 27
  17. 2025Route Sparse Autoencoder to Interpret Large Language Modelscited by 25
  18. 2025Interpretable and Testable Vision Features via Sparse Autoencoderscited by 24
  19. 2025Narrow Finetuning Leaves Clearly Readable Traces in Activation Differencescited by 24
  20. 2021Translation Error Detection as Rationale Extractioncited by 22
  21. 2024Automatically Identifying Local and Global Circuits with Linear Computation Graphscited by 22
  22. 2025SAE-V: Interpreting Multimodal Models for Enhanced Alignmentcited by 22
  23. 2022How to Dissect a Muppet: The Structure of Transformer Embedding Spacescited by 19
  24. 2024PaCE: Parsimonious Concept Engineering for Large Language Modelscited by 19
  25. 2024Residual Stream Analysis with Multi-Layer SAEscited by 18
  26. 2025Dense SAE Latents Are Features, Not Bugscited by 17
  27. 2025From superposition to sparse codes: interpretable representations in neural networkscited by 16
  28. 2025TIDE : Temporal-Aware Sparse Autoencoders for Interpretable Diffusion Transformers in Image Generationcited by 16
  29. 2024Decoding Dark Matter: Specialized Sparse Autoencoders for Interpreting Rare Concepts in Foundation Modelscited by 15
  30. 2025Stochastic Parameter Decompositioncited by 14
  31. 2024A Complexity-Based Theory of Compositionalitycited by 13
  32. 2025Diversity-driven Data Selection for Language Model Tuning through Sparse Autoencodercited by 11
  33. 2025The Dead Salmons of AI Interpretabilitycited by 10
  34. 2026From Directions to Regions: Decomposing Activations in Language Models via Local Geometrycited by 10
  35. 2025Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activationscited by 9
  36. 2025Do Sparse Autoencoders Generalize? A Case Study of Answerabilitycited by 8
  37. 2025Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Modelscited by 8
  38. 2024Towards Compositionality in Concept Learningcited by 7

50 most cited of 106 All 106 (JSON)