mlsanityData to 7 Oct 2026

InversionView: A General-Purpose Method for Reading Information from Neural Activations

arXiv 2405.17653 ↗First posted 27 May 2024cs.LG cs.AI cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

The inner workings of neural networks can be better understood if we can fully decipher the information encoded in neural activations. In this paper, we argue that this information is embodied by the subset of inputs that give rise to similar activations. We propose InversionView, which allows us to practically inspect this subset by sampling from a trained decoder model conditioned on activations. This helps uncover the information content of activation vectors, and facilitates understanding of the algorithms implemented by transformer models. We present four case studies where we investigate models ranging from small transformers to GPT-2. In these studies, we show that InversionView can reveal clear information contained in activations, including basic information about tokens appearing in the context, as well as more complex information, such as the count of certain tokens, their relative positions, and abstract knowledge about the subject. We also provide causally verified circuits to confirm the decoded information.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 30 of its 39 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

30 of 39 references.

References · 30

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  4. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  5. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  6. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  7. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  8. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  9. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  10. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  11. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  12. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
18 more
  1. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  2. 2021Causal Abstractions of Neural Networkscited by 401
  3. 2018Analysis Methods in Neural Language Processing: A Surveycited by 276
  4. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 225
  5. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  6. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 194
  7. 2023Localizing Model Behavior with Path Patchingcited by 182
  8. 2021Implicit Representations of Meaning in Neural Language Modelscited by 179
  9. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 165
  10. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 145
  11. 2022Analyzing Transformers in Embedding Spacecited by 131
  12. 2023Future Lens: Anticipating Subsequent Tokens from a Single Hidden Statecited by 116
  13. 2023Interpretability at Scale: Identifying Causal Mechanisms in Alpacacited by 102
  14. 2023A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysiscited by 88
  15. 2023Codebook Features: Sparse and Discrete Interpretability for Neural Networkscited by 45
  16. 2023Understanding Addition in Transformerscited by 34
  17. 2023VISIT: Visualizing and Interpreting the Semantic Information Flow of Transformerscited by 32
  18. 2023Gaussian Process Probes (GPP) for Uncertainty-Aware Probingcited by 11

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 5.

7 papers.

  1. 2025Decomposing Representation Space into Interpretable Subspaces with Unsupervised Learningcited by 7
  2. 2025Rep2Text: Decoding Full Text from a Single LLM Token Representationcited by 7
  3. 2024Eliciting Textual Descriptions from Representations of Continuous Promptscited by 3
  4. 2025Pierce the Mists, Greet the Sky: Decipher Knowledge Overshadowing via Knowledge Circuit Analysiscited by 3
  5. 2025InverseScope: Scalable Activation Inversion for Interpreting Large Language Modelscited by 2
  6. 2025GPT, But Backwards: Exactly Inverting Language Model Outputscited by 2
  7. 2024Identifying Sub-networks in Neural Networks via Functionally Similar Representationscited by 0