# Backward Lens: Projecting Language Model Gradients into the Vocabulary Space

arXiv: https://arxiv.org/abs/2402.12865 · HTML: https://mlsanity.com/paper/2402.12865/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 53 references that are in the map.
- First posted on arXiv 20 Feb 2024.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space](https://mlsanity.com/paper/2203.14680/) (2022)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [Analysis Methods in Neural Language Processing: A Survey](https://mlsanity.com/paper/1812.08951/) (2018)
- [Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning](https://mlsanity.com/paper/2305.14160/) (2023)
- [Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models](https://mlsanity.com/paper/2401.06102/) (2024)
- [Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models](https://mlsanity.com/paper/1708.08296/) (2017)
- [Linear Representations of Sentiment in Large Language Models](https://mlsanity.com/paper/2310.15154/) (2023)
- [Visual Interpretability for Deep Learning: a Survey](https://mlsanity.com/paper/1802.00614/) (2018)
- [Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformer](https://mlsanity.com/paper/2305.16380/) (2023)
- [Jump to Conclusions: Short-Cutting Transformers With Linear Transformations](https://mlsanity.com/paper/2303.09435/) (2023)
- [Inseq: An Interpretability Toolkit for Sequence Generation Models](https://mlsanity.com/paper/2302.13942/) (2023)
- [Understanding Transformer Memorization Recall Through Idioms](https://mlsanity.com/paper/2210.03588/) (2022)
- [Knowledge is a Region in Weight Space for Fine-tuned Language Models](https://mlsanity.com/paper/2302.04863/) (2023)
- [Codebook Features: Sparse and Discrete Interpretability for Neural Networks](https://mlsanity.com/paper/2310.17230/) (2023)
- [Discretized Integrated Gradients for Explaining Language Models](https://mlsanity.com/paper/2108.13654/) (2021)

## Built on by (most cited first, 30 of 32)

- [A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models](https://mlsanity.com/paper/2407.02646/) (2024)
- [SAEs Are Good for Steering -- If You Select the Right Features](https://mlsanity.com/paper/2505.20063/) (2025)
- [Verbalizable Representations Form a Global Workspace in Language Models](https://mlsanity.com/paper/2607.15495/) (2026)
- [Mechanistic Understanding and Mitigation of Language Model Non-Factual Hallucinations](https://mlsanity.com/paper/2403.18167/) (2024)
- [Activation Scaling for Steering and Interpreting Language Models](https://mlsanity.com/paper/2410.04962/) (2024)
- [Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models](https://mlsanity.com/paper/2601.14004/) (2026)
- [LogitLens4LLMs: Extending Logit Lens Analysis to Modern Large Language Models](https://mlsanity.com/paper/2503.11667/) (2025)
- [Token Erasure as a Footprint of Implicit Vocabulary Items in LLMs](https://mlsanity.com/paper/2406.20086/) (2024)
- [Inferring Functionality of Attention Heads from their Parameters](https://mlsanity.com/paper/2412.11965/) (2024)
- [Dissecting Bias in LLMs: A Mechanistic Interpretability Perspective](https://mlsanity.com/paper/2506.05166/) (2025)
- [Attributing Response to Context: A Jensen-Shannon Divergence Driven Mechanistic Study of Context Attribution in Retrieval-Augmented Generation](https://mlsanity.com/paper/2505.16415/) (2025)
- [LLM Microscope: What Model Internals Reveal About Answer Correctness and Context Utilization](https://mlsanity.com/paper/2510.04013/) (2025)
- [Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safety](https://mlsanity.com/paper/2506.05451/) (2025)
- [Fine-Grained Activation Steering: Steering Less, Achieving More](https://mlsanity.com/paper/2602.04428/) (2026)
- [Addition in Four Movements: Mapping Layer-wise Information Trajectories in LLMs](https://mlsanity.com/paper/2506.07824/) (2025)
- [Enhancing Conversational Agents with Theory of Mind: Aligning Beliefs, Desires, and Intentions for Human-Like Interaction](https://mlsanity.com/paper/2502.14171/) (2025)
- [TensorLens: End-to-End Transformer Analysis via High-Order Attention Tensors](https://mlsanity.com/paper/2601.17958/) (2026)
- [Reversed Attention: On The Gradient Descent Of Attention Layers In GPT](https://mlsanity.com/paper/2412.17019/) (2024)
- [LLM Braces: Straightening Out LLM Predictions with Relevant Sub-Updates](https://mlsanity.com/paper/2503.16334/) (2025)
- [LoKI: Low-damage Knowledge Implanting of Large Language Models](https://mlsanity.com/paper/2505.22120/) (2025)
- [Probability Signature: Bridging Data Semantics and Embedding Structure in Language Models](https://mlsanity.com/paper/2509.20124/) (2025)
- [Vocabulary embeddings organize linguistic structure early in language model training](https://mlsanity.com/paper/2510.07613/) (2025)
- [Pitfalls in Evaluating Interpretability Agents](https://mlsanity.com/paper/2603.20101/) (2026)
- [Understanding Fact Recall in Language Models: Why Two-Stage Training Encourages Memorization but Mixed Training Teaches Knowledge](https://mlsanity.com/paper/2505.16178/) (2025)
- [Latent Trajectory Dynamics in Large Language Models: A Manifold Evolution Framework with Empirical Validation](https://mlsanity.com/paper/2505.20340/) (2025)
- [Building Reasonable Inference for Vision-Language Models in Blind Image Quality Assessment](https://mlsanity.com/paper/2512.09555/) (2025)
- [Factual Retrieval in LLMs Is a Redundant, Distributed and Non-Contiguous Process](https://mlsanity.com/paper/2606.21345/) (2026)
- [Check Your LLM's Secret Dictionary! Five Lines of Code Reveal What Your LLM Learned (Including What It Shouldn't Have)](https://mlsanity.com/paper/2605.22005/) (2026)
- [Query Lens: Interpreting Sparse Key-Value Features with Indirect Effects](https://mlsanity.com/paper/2606.07617/) (2026)
- [Interpreting Language Model Hidden States at Scale](https://mlsanity.com/paper/2608.10260/) (2026)

## Abstract (arXiv)

Understanding how Transformer-based Language Models (LMs) learn and recall information is a key goal of the deep learning community. Recent interpretability methods project weights and hidden states obtained from the forward pass to the models' vocabularies, helping to uncover how information flows within LMs. In this work, we extend this methodology to LMs' backward pass and gradients. We first prove that a gradient matrix can be cast as a low-rank linear combination of its forward and backward passes' inputs. We then develop methods to project these gradients into vocabulary items and explore the mechanics of how new information is stored in the LMs' neurons.