Measuring Feature Sparsity in Language Models
Not on the map. Why?
Abstract (arXiv)
Recent works have proposed that activations in language models can be modelled as sparse linear combinations of vectors corresponding to features of input text. Under this assumption, these works aimed to reconstruct feature directions using sparse coding. We develop metrics to assess the success of these sparse coding techniques and test the validity of the linearity and sparsity assumptions. We show our metrics can predict the level of sparsity on synthetic sparse linear activations, and can distinguish between sparse linear data and several other distributions. We use our metrics to measure levels of sparsity in several language models. We find evidence that language model activations can be accurately modelled by sparse linear combinations of features, significantly more so than control datasets. We also show that model activations appear to be sparsest in the first and final layers.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 30 of its 50 references that are in the map. First posted on arXiv 11 Oct 2023.
Builds on
30 of 50 references.
References · 30
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2023GPT-4 Technical Reportcited by 18,396
- 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
- 2022Locating and Editing Factual Associations in GPTcited by 2,691
- 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
- 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
- 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
- 2022Editing Models with Task Arithmeticcited by 1,120
- 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
- 2022YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectorscited by 982
- 2022Toy Models of Superpositioncited by 971
- 2023Steering Language Models With Activation Engineeringcited by 922
18 more
- 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
- 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
- 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
- 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
- 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
- 2021Causal Abstractions of Neural Networkscited by 422
- 2016Linear Algebraic Structure of Word Senses, with Applications to Polysemycited by 165
- 2017Optimizing the Latent Space of Generative Networkscited by 157
- 2022CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networkscited by 147
- 2019Well-Read Students Learn Better: On the Importance of Pre-training Compact Modelscited by 120
- 2021Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factorscited by 108
- 2022Linear Adversarial Concept Erasurecited by 101
- 2022Interpreting Language Models with Contrastive Explanationscited by 92
- 2018Understanding and Improving Interpolation in Autoencoders via an Adversarial Regularizercited by 88
- 2022Polysemanticity and Capacity in Neural Networkscited by 77
- 2021Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristicscited by 67
- 2015Sparse Overcomplete Word Vector Representationscited by 44
- 2021Interpreting the Latent Space of Generative Adversarial Networks using Supervised Learningcited by 4
Built on by
Citing papers per year: 2024: 1; 2025: 1; 2026: 1.
3 papers.