# Semantic Structure and Interpretability of Word Embeddings

arXiv: https://arxiv.org/abs/1711.00331 · HTML: https://mlsanity.com/paper/1711.00331/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Nov 2017.

## Built on by (most cited first, 21 of 21)

- [Word Embeddings Are Steers for Language Models](https://mlsanity.com/paper/2305.12798/) (2023)
- [DirectProbe: Studying Representations without Classifiers](https://mlsanity.com/paper/2104.05904/) (2021)
- [Interpreting Deep Learning Models in Natural Language Processing: A Review](https://mlsanity.com/paper/2110.10470/) (2021)
- [A Crosslingual Investigation of Conceptualization in 1335 Languages](https://mlsanity.com/paper/2305.08475/) (2023)
- [Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaning](https://mlsanity.com/paper/2506.08354/) (2025)
- [Explainable Semantic Space by Grounding Language to Vision with Cross-Modal Contrastive Learning](https://mlsanity.com/paper/2111.07180/) (2021)
- [DINE: Dimensional Interpretability of Node Embeddings](https://mlsanity.com/paper/2310.01162/) (2023)
- [Deep Transfer Learning & Beyond: Transformer Language Models in Information Systems Research](https://mlsanity.com/paper/2110.08975/) (2021)
- [SWEA: Updating Factual Knowledge in Large Language Models via Subject Word Embedding Altering](https://mlsanity.com/paper/2401.17809/) (2024)
- [Interpreting Embedding Spaces by Conceptualization](https://mlsanity.com/paper/2209.00445/) (2022)
- [Quantifying Positional Biases in Text Embedding Models](https://mlsanity.com/paper/2412.15241/) (2024)
- [Disentangling Linguistic Features with Dimension-Wise Analysis of Vector Embeddings](https://mlsanity.com/paper/2504.14766/) (2025)
- [Rethinking Caching for LLM Serving Systems: Beyond Traditional Heuristics](https://mlsanity.com/paper/2508.18736/) (2025)
- [Character-level Chinese Backpack Language Models](https://mlsanity.com/paper/2310.12751/) (2023)
- [Disentangled and Self-Explainable Node Representation Learning](https://mlsanity.com/paper/2410.21043/) (2024)
- [Explanations Go Linear: Post-hoc Explainability for Tabular Data with Interpretable Meta-Encoding](https://mlsanity.com/paper/2504.20667/) (2025)
- [Lex2vec: making Explainable Word Embeddings via Lexical Resources](https://mlsanity.com/paper/2103.02269/) (2021)
- [Shape of Elephant: Study of Macro Properties of Word Embeddings Spaces](https://mlsanity.com/paper/2106.06964/) (2021)
- [Structural invariants and semantic fingerprints in the "ego network" of words](https://mlsanity.com/paper/2203.00588/) (2022)
- [Contrastive Learning as Kernel Approximation](https://mlsanity.com/paper/2309.02651/) (2023)
- [Optimal synthesis embeddings](https://mlsanity.com/paper/2406.10259/) (2024)

## Abstract (arXiv)

Dense word embeddings, which encode semantic meanings of words to low dimensional vector spaces have become very popular in natural language processing (NLP) research due to their state-of-the-art performances in many NLP tasks. Word embeddings are substantially successful in capturing semantic relations among words, so a meaningful semantic structure must be present in the respective vector spaces. However, in many cases, this semantic structure is broadly and heterogeneously distributed across the embedding dimensions, which makes interpretation a big challenge. In this study, we propose a statistical method to uncover the latent semantic structure in the dense word embeddings. To perform our analysis we introduce a new dataset (SEMCAT) that contains more than 6500 words semantically grouped under 110 categories. We further propose a method to quantify the interpretability of the word embeddings; the proposed method is a practical alternative to the classical word intrusion test that requires human intervention.