# Towards Understanding the Nature of Attention with Low-Rank Sparse Decomposition arXiv: https://arxiv.org/abs/2504.20938 ยท HTML: https://mlsanity.com/paper/2504.20938/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability. - Builds on 28 of its 43 references that are in the map. - First posted on arXiv 29 Apr 2025. ## Placed in - Mechanistic interpretability / [Circuits and mechanistic analysis](https://mlsanity.com/mechinterp/circuits-and-mechanistic-analysis/) ## Builds on (most cited first, 28 of 28) - [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021) - [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022) - [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023) - [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023) - [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023) - [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023) - [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022) - [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022) - [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023) - [Scaling and evaluating sparse autoencoders](https://mlsanity.com/paper/2406.04093/) (2024) - [Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2](https://mlsanity.com/paper/2408.05147/) (2024) - [Towards Best Practices of Activation Patching in Language Models: Metrics and Methods](https://mlsanity.com/paper/2309.16042/) (2023) - [Finding Neurons in a Haystack: Case Studies with Sparse Probing](https://mlsanity.com/paper/2305.01610/) (2023) - [Retrieval Head Mechanistically Explains Long-Context Factuality](https://mlsanity.com/paper/2404.15574/) (2024) - [How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model](https://mlsanity.com/paper/2305.00586/) (2023) - [Transcoders Find Interpretable LLM Feature Circuits](https://mlsanity.com/paper/2406.11944/) (2024) - [Improving Dictionary Learning with Gated Sparse Autoencoders](https://mlsanity.com/paper/2404.16014/) (2024) - [Linear Algebraic Structure of Word Senses, with Applications to Polysemy](https://mlsanity.com/paper/1601.03764/) (2016) - [Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders](https://mlsanity.com/paper/2410.20526/) (2024) - [Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics](https://mlsanity.com/paper/2410.21272/) (2024) - [Successor Heads: Recurring, Interpretable Attention Heads In The Wild](https://mlsanity.com/paper/2312.09230/) (2023) - [Copy Suppression: Comprehensively Understanding an Attention Head](https://mlsanity.com/paper/2310.04625/) (2023) - [Decomposing The Dark Matter of Sparse Autoencoders](https://mlsanity.com/paper/2410.14670/) (2024) - [Interpreting Attention Layer Outputs with Sparse Autoencoders](https://mlsanity.com/paper/2406.17759/) (2024) - [Towards Universality: Studying Mechanistic Similarity Across Language Model Architectures](https://mlsanity.com/paper/2410.06672/) (2024) - [Dictionary Learning Improves Patch-Free Circuit Discovery in Mechanistic Interpretability: A Case Study on Othello-GPT](https://mlsanity.com/paper/2402.12201/) (2024) - [An X-Ray Is Worth 15 Features: Sparse Autoencoders for Interpretable Radiology Report Generation](https://mlsanity.com/paper/2410.03334/) (2024) - [Automatically Identifying Local and Global Circuits with Linear Computation Graphs](https://mlsanity.com/paper/2405.13868/) (2024) ## Built on by (most cited first, 9 of 9) - [Adversarial Attacks Leverage Interference Between Features in Superposition](https://mlsanity.com/paper/2510.11709/) (2025) - [Dimensional Collapse in Transformer Attention Outputs: A Challenge for Sparse Dictionary Learning](https://mlsanity.com/paper/2508.16929/) (2025) - [Tracing the Thought of a Grandmaster-level Chess-Playing Transformer](https://mlsanity.com/paper/2604.10158/) (2026) - [Transcoder Adapters for Reasoning-Model Diffing](https://mlsanity.com/paper/2602.20904/) (2026) - [The Past Is Not Past: Memory-Enhanced Dynamic Reward Shaping](https://mlsanity.com/paper/2604.11297/) (2026) - [WriteSAE: Sparse Autoencoders for Recurrent State](https://mlsanity.com/paper/2605.12770/) (2026) - [A Distributional View for Visual Mechanistic Interpretability: KL-Minimal Soft-Constraint Principle](https://mlsanity.com/paper/2605.17504/) (2026) - [Targeted Recovery of Weight-Space Mechanisms From Neural Networks](https://mlsanity.com/paper/2607.13047/) (2026) - [Weights Read and Write Features: Scalable Parameter Decomposition Grounded in Activation Space](https://mlsanity.com/paper/2609.37731/) (2026) ## Abstract (arXiv) We propose Low-Rank Sparse Attention (Lorsa), a sparse replacement model of Transformer attention layers to disentangle original Multi Head Self Attention (MHSA) into individually comprehensible components. Lorsa is designed to address the challenge of attention superposition to understand attention-mediated interaction between features in different token positions. We show that Lorsa heads find cleaner and finer-grained versions of previously discovered MHSA behaviors like induction heads, successor heads and attention sink behavior (i.e., heavily attending to the first token). Lorsa and Sparse Autoencoder (SAE) are both sparse dictionary learning methods applied to different Transformer components, and lead to consistent findings in many ways. For instance, we discover a comprehensive family of arithmetic-specific Lorsa heads, each corresponding to an atomic operation in Llama-3.1-8B. Automated interpretability analysis indicates that Lorsa achieves parity with SAE in interpretability while Lorsa exhibits superior circuit discovery properties, especially for features computed collectively by multiple MHSA heads. We also conduct extensive experiments on architectural design ablation, Lorsa scaling law and error analysis.