mlsanityData to 7 Oct 2026

Backward Lens: Projecting Language Model Gradients into the Vocabulary Space

arXiv 2402.12865 ↗First posted 20 Feb 2024cs.CL cs.AI cs.LGcited by 30

Not on the map. Why?

Abstract (arXiv)

Understanding how Transformer-based Language Models (LMs) learn and recall information is a key goal of the deep learning community. Recent interpretability methods project weights and hidden states obtained from the forward pass to the models' vocabularies, helping to uncover how information flows within LMs. In this work, we extend this methodology to LMs' backward pass and gradients. We first prove that a gradient matrix can be cast as a low-rank linear combination of its forward and backward passes' inputs. We then develop methods to project these gradients into vocabulary items and explore the mechanics of how new information is stored in the LMs' neurons.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 35 of its 53 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

35 of 53 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2016Layer Normalizationcited by 5,579
  6. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  7. 2016Pointer Sentinel Mixture Modelscited by 2,992
  8. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  9. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  11. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  12. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
23 more
  1. 2022Mass-Editing Memory in a Transformercited by 976
  2. 2022Toy Models of Superpositioncited by 928
  3. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  4. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  5. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  6. 2018Analysis Methods in Neural Language Processing: A Surveycited by 276
  7. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  8. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 225
  9. 2017Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Modelscited by 202
  10. 2023Linear Representations of Sentiment in Large Language Modelscited by 157
  11. 2018Visual Interpretability for Deep Learning: a Surveycited by 155
  12. 2023Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformercited by 101
  13. 2023Jump to Conclusions: Short-Cutting Transformers With Linear Transformationscited by 89
  14. 2023Inseq: An Interpretability Toolkit for Sequence Generation Modelscited by 78
  15. 2022Understanding Transformer Memorization Recall Through Idiomscited by 62
  16. 2023Knowledge is a Region in Weight Space for Fine-tuned Language Modelscited by 58
  17. 2021Discretized Integrated Gradients for Explaining Language Modelscited by 45
  18. 2023Codebook Features: Sparse and Discrete Interpretability for Neural Networkscited by 45
  19. 2022Optimizing Relevance Maps of Vision Transformers Improves Robustnesscited by 41
  20. 2023Using Captum to Explain Generative Language Modelscited by 33
  21. 2023VISIT: Visualizing and Interpreting the Semantic Information Flow of Transformerscited by 32
  22. 2022What Are You Token About? Dense Retrieval as Distributions Over the Vocabularycited by 28
  23. 2022projUNN: efficient method for training deep networks with unitary matricescited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 15; 2026: 9.

30 papers.

Sort:

By area:

  1. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  2. 2025SAEs Are Good for Steering -- If You Select the Right Featurescited by 74
  3. 2024Mechanistic Understanding and Mitigation of Language Model Non-Factual Hallucinationscited by 38
  4. 2024Activation Scaling for Steering and Interpreting Language Modelscited by 33
  5. 2025LogitLens4LLMs: Extending Logit Lens Analysis to Modern Large Language Modelscited by 22
  6. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 22
  7. 2024Token Erasure as a Footprint of Implicit Vocabulary Items in LLMscited by 17
  8. 2024Inferring Functionality of Attention Heads from their Parameterscited by 17
  9. 2025Dissecting Bias in LLMs: A Mechanistic Interpretability Perspectivecited by 14
  10. 2025Attributing Response to Context: A Jensen-Shannon Divergence Driven Mechanistic Study of Context Attribution in Retrieval-Augmented Generationcited by 12
  11. 2025LLM Microscope: What Model Internals Reveal About Answer Correctness and Context Utilizationcited by 11
  12. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
18 more
  1. 2026Fine-Grained Activation Steering: Steering Less, Achieving Morecited by 6
  2. 2025Enhancing Conversational Agents with Theory of Mind: Aligning Beliefs, Desires, and Intentions for Human-Like Interactioncited by 3
  3. 2025Addition in Four Movements: Mapping Layer-wise Information Trajectories in LLMscited by 3
  4. 2024Reversed Attention: On The Gradient Descent Of Attention Layers In GPTcited by 2
  5. 2025LLM Braces: Straightening Out LLM Predictions with Relevant Sub-Updatescited by 2
  6. 2025LoKI: Low-damage Knowledge Implanting of Large Language Modelscited by 2
  7. 2025Probability Signature: Bridging Data Semantics and Embedding Structure in Language Modelscited by 2
  8. 2026Pitfalls in Evaluating Interpretability Agentscited by 2
  9. 2025Understanding Fact Recall in Language Models: Why Two-Stage Training Encourages Memorization but Mixed Training Teaches Knowledgecited by 1
  10. 2025Latent Trajectory Dynamics in Large Language Models: A Manifold Evolution Framework with Empirical Validationcited by 1
  11. 2025Vocabulary embeddings organize linguistic structure early in language model trainingcited by 1
  12. 2025Building Reasonable Inference for Vision-Language Models in Blind Image Quality Assessmentcited by 1
  13. 2026TensorLens: End-to-End Transformer Analysis via High-Order Attention Tensorscited by 1
  14. 2026Factual Retrieval in LLMs Is a Redundant, Distributed and Non-Contiguous Processcited by 1
  15. 2026Check Your LLM's Secret Dictionary! Five Lines of Code Reveal What Your LLM Learned (Including What It Shouldn't Have)cited by 0
  16. 2026Query Lens: Interpreting Sparse Key-Value Features with Indirect Effectscited by 0
  17. 2026Interpreting Language Model Hidden States at Scalecited by 0
  18. 2026Using OCR Heads to Verbalize Image Semanticscited by 0