mlsanityData to 7 Oct 2026

Sparse Overcomplete Word Vector Representations

arXiv 1506.02004 ↗First posted 5 Jun 2015cs.CLcited by 43

Not on the map. Why?

Abstract (arXiv)

Current distributed representations of words show little resemblance to theories of lexical semantics. The former are dense and uninterpretable, the latter largely based on familiar, discrete classes (e.g., supersenses) and relations (e.g., synonymy and hypernymy). We propose methods that transform word vectors into sparse (and optionally binary) vectors. The resulting representations are more similar to the interpretable features typically used in NLP, though they are discovered automatically from raw corpora. Because the vectors are highly sparse, they are computationally easy to work with. Most importantly, we find that they outperform the original vectors on benchmark tasks.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Jun 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 3; 2023: 7; 2024: 7; 2025: 17; 2026: 1.

43 papers.

Sort:

By area:

  1. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  2. 2024Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)cited by 116
  3. 2021Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factorscited by 106
  4. 2024Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Controlcited by 83
  5. 2021Neuron-level Interpretation of Deep NLP Models: A Surveycited by 81
  6. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Modelscited by 73
  7. 2024Identifying Functionally Important Features with End-to-End Sparse Dictionary Learningcited by 64
  8. 2025Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometrycited by 58
  9. 2021A Survey on Green Deep Learningcited by 52
  10. 2024The Geometry of Concepts: Sparse Autoencoder Feature Structurecited by 52
  11. 2022On the Explainability of Natural Language Processing Deep Modelscited by 37
  12. 2024FluxSpace: Disentangled Semantic Editing in Rectified Flow Transformerscited by 25
31 more
  1. 2022Simplicial Embeddings in Self-Supervised Learning and Downstream Classificationcited by 23
  2. 2023Discovering Universal Geometry in Embeddings with ICAcited by 19
  3. 2025Transferring Linear Features Across Language Models With Model Stitchingcited by 17
  4. 2025Dense SAE Latents Are Features, Not Bugscited by 17
  5. 2025From superposition to sparse codes: interpretable representations in neural networkscited by 16
  6. 2023Tsetlin Machine Embedding: Representing Words Using Logical Expressionscited by 15
  7. 2025Refusal Behavior in Large Language Models: A Nonlinear Perspectivecited by 13
  8. 2025Interpretable Text Embeddings and Text Similarity Explanation: A Surveycited by 12
  9. 2025CorrSteer: Generation-Time LLM Steering via Correlated Sparse Autoencoder Featurescited by 10
  10. 2025SAGE: An Agentic Explainer Framework for Interpreting SAE Features in Language Modelscited by 10
  11. 2023SensePOLAR: Word sense aware interpretability for pre-trained contextual word embeddingscited by 9
  12. 2023DINE: Dimensional Interpretability of Node Embeddingscited by 8
  13. 2021Ultra-High Dimensional Sparse Representations with Binarization for Efficient Text Retrievalcited by 7
  14. 2025Quantifying Structure in CLIP Embeddings: A Statistical Framework for Concept Interpretationcited by 6
  15. 2024Revisiting Cosine Similarity via Normalized ICA-transformed Embeddingscited by 5
  16. 2025BehaviorBox: Automated Discovery of Fine-Grained Performance Differences Between Language Modelscited by 5
  17. 2021Biomedical Interpretable Entity Representationscited by 4
  18. 2021Extending Multi-Sense Word Embedding to Phrases and Sentences for Unsupervised Semantic Applicationscited by 3
  19. 2023Measuring Feature Sparsity in Language Modelscited by 3
  20. 2021Interpretable contrastive word mover's embeddingcited by 2
  21. 2025Disentangling Linguistic Features with Dimension-Wise Analysis of Vector Embeddingscited by 2
  22. 2025Geometry of Semantics in Next-Token Prediction: How Optimization Implicitly Organizes Linguistic Representationscited by 2
  23. 2025FaithfulSAE: Towards Capturing Faithful Features with Sparse Autoencoders without External Dataset Dependenciescited by 2
  24. 2025Exploring Task Performance with Interpretable Models via Sparse Auto-Encoderscited by 1
  25. 2021SEMIE: SEMantically Infused Embeddings with Enhanced Interpretability for Domain-specific Small Corpuscited by 0
  26. 2022Emergent organization of receptive fields in networks of excitatory and inhibitory neuronscited by 0
  27. 2023Interpretable Neural Embeddings with Sparse Self-Representationcited by 0
  28. 2024Interpretable Syntactic Representations Enable Hierarchical Word Vectorscited by 0
  29. 2025Dictionary Learning: The Complexity of Learning Sparse Superposed Features with Feedbackcited by 0
  30. 2025Analysis of Variational Sparse Autoencoderscited by 0
  31. 2026When Numbers Tell Half the Story: Human-Metric Alignment in Topic Model Evaluationcited by 0