# Word Embedding Visualization Via Dictionary Learning

arXiv: https://arxiv.org/abs/1910.03833 · HTML: https://mlsanity.com/paper/1910.03833/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Oct 2019.

## Built on by (most cited first, 12 of 12)

- [Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet](https://mlsanity.com/paper/2605.29358/) (2026)
- [Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors](https://mlsanity.com/paper/2103.15949/) (2021)
- [The Geometry of Concepts: Sparse Autoencoder Feature Structure](https://mlsanity.com/paper/2410.19750/) (2024)
- [Dictionary Learning Improves Patch-Free Circuit Discovery in Mechanistic Interpretability: A Case Study on Othello-GPT](https://mlsanity.com/paper/2402.12201/) (2024)
- [Interpretable and Testable Vision Features via Sparse Autoencoders](https://mlsanity.com/paper/2502.06755/) (2025)
- [Automatically Identifying Local and Global Circuits with Linear Computation Graphs](https://mlsanity.com/paper/2405.13868/) (2024)
- [Dense SAE Latents Are Features, Not Bugs](https://mlsanity.com/paper/2506.15679/) (2025)
- [From superposition to sparse codes: interpretable representations in neural networks](https://mlsanity.com/paper/2503.01824/) (2025)
- [Minimalistic Unsupervised Learning with the Sparse Manifold Transform](https://mlsanity.com/paper/2209.15261/) (2022)
- [Geometry of Semantics in Next-Token Prediction: How Optimization Implicitly Organizes Linguistic Representations](https://mlsanity.com/paper/2505.08348/) (2025)
- [Dictionary Learning: The Complexity of Learning Sparse Superposed Features with Feedback](https://mlsanity.com/paper/2502.05407/) (2025)
- [Analysis of Variational Sparse Autoencoders](https://mlsanity.com/paper/2509.22994/) (2025)

## Abstract (arXiv)

Co-occurrence statistics based word embedding techniques have proved to be very useful in extracting the semantic and syntactic representation of words as low dimensional continuous vectors. In this work, we discovered that dictionary learning can open up these word vectors as a linear combination of more elementary word factors. We demonstrate many of the learned factors have surprisingly strong semantic or syntactic meaning corresponding to the factors previously identified manually by human inspection. Thus dictionary learning provides a powerful visualization tool for understanding word embedding representations. Furthermore, we show that the word factors can help in identifying key semantic and syntactic differences in word analogy tasks and improve upon the state-of-the-art word embedding techniques in these tasks by a large margin.