mlsanityData to 14 Oct 2026

LM Transparency Tool: Interactive Tool for Analyzing Transformer Language Models

arXiv 2404.07004 ↗First posted 10 Apr 2024cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

We present the LM Transparency Tool (LM-TT), an open-source interactive toolkit for analyzing the internal workings of Transformer-based language models. Differently from previously existing tools that focus on isolated parts of the decision-making process, our framework is designed to make the entire prediction process transparent, and allows tracing back model behavior from the top-layer representation to very fine-grained parts of the model. Specifically, it (1) shows the important part of the whole input-to-output information flow, (2) allows attributing any changes done by a model block to individual attention heads and feed-forward neurons, (3) allows interpreting the functions of those heads or neurons. A crucial part of this pipeline is showing the importance of specific model components at each step. As a result, we are able to look at the roles of model components only in cases where they are important for a prediction. Since knowing which components should be inspected is key for analyzing large models where the number of these components is extremely high, we believe our tool will greatly support the interpretability community both in research settings and in practical applications.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 26 of its 41 references that are in the map. First posted on arXiv 10 Apr 2024.

Builds on

26 of 41 references.

References · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  7. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  8. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  9. 2022In-context Learning and Induction Headscited by 964
  10. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  11. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  12. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
14 more
  1. 2020Captum: A unified and generic model interpretability library for PyTorchcited by 547
  2. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  3. 2021Causal Abstractions of Neural Networkscited by 422
  4. 2019Adaptively Sparse Transformerscited by 155
  5. 2022Analyzing Transformers in Embedding Spacecited by 132
  6. 2020The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Modelscited by 125
  7. 2020The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?cited by 122
  8. 2024Information Flow Routes: Automatically Interpreting Language Models at Scalecited by 95
  9. 2023Jump to Conclusions: Short-Cutting Transformers With Linear Transformationscited by 90
  10. 2023Neurons in Large Language Models: Dead, N-gram, Positionalcited by 88
  11. 2023Inseq: An Interpretability Toolkit for Sequence Generation Modelscited by 78
  12. 2022Measuring the Mixing of Contextual Information in the Transformercited by 65
  13. 2023Using Captum to Explain Generative Language Modelscited by 33
  14. 2023VISIT: Visualizing and Interpreting the Semantic Information Flow of Transformerscited by 32

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 6; 2026: 3.

16 papers.

Sort:

By area:

  1. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 85
  2. 2024Finding Transformer Circuits with Edge Pruningcited by 60
  3. 2024Navigating LLM Ethics: Advancements, Challenges, and Future Directionscited by 28
  4. 2024The Same But Different: Structural Similarities and Differences in Multilingual Language Modelingcited by 28
  5. 2025ReLearn: Unlearning via Learning for Large Language Modelscited by 22
  6. 2024Unconditional Truthfulness: Learning Unconditional Uncertainty of Large Language Modelscited by 9
  7. 2024CoXQL: A Dataset for Parsing Explanation Requests in Conversational XAI Systemscited by 5
  8. 2025Function Induction and Task Generalization: An Interpretability Study with Off-by-One Additioncited by 2
  9. 2024Analyzing Context Contributions in LLM-based Machine Translationcited by 1
  10. 2025Know-MRI: A Knowledge Mechanisms Revealer&Interpreter for Large Language Modelscited by 1
  11. 2025ConceptViz: A Visual Analytics Approach for Exploring Concepts in Large Language Modelscited by 1
  12. 2025Char-mander Use mBackdoor! A Study of Cross-lingual Backdoor Attacks in Multilingual LLMscited by 0
4 more
  1. 2025InTraVisTo: Inside Transformer Visualisation Toolcited by 0
  2. 2026Simplifying Outcomes of Language Model Component Analyses with ELIAcited by 0
  3. 2026VeriLLMed: Interactive Visual Debugging of Medical Large Language Models with Knowledge Graphscited by 0
  4. 2026LLM-FACETS: A Privacy-Preserving Framework for Evaluating LLM Transparency and Accountabilitycited by 0