# Interpretable Syntactic Representations Enable Hierarchical Word Vectors

arXiv: https://arxiv.org/abs/2411.08384 · HTML: https://mlsanity.com/paper/2411.08384/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 31 references that are in the map.
- First posted on arXiv 13 Nov 2024.

## Builds on (most cited first, 7 of 7)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [All-but-the-Top: Simple and Effective Postprocessing for Word Representations](https://mlsanity.com/paper/1702.01417/) (2017)
- [Sparse Overcomplete Word Vector Representations](https://mlsanity.com/paper/1506.02004/) (2015)
- [WizMap: Scalable Interactive Visualization for Exploring Large Machine Learning Embeddings](https://mlsanity.com/paper/2306.09328/) (2023)
- [Problems With Evaluation of Word Embeddings Using Word Similarity Tasks](https://mlsanity.com/paper/1605.02276/) (2016)
- [SensePOLAR: Word sense aware interpretability for pre-trained contextual word embeddings](https://mlsanity.com/paper/2301.04704/) (2023)
- [The Secret is in the Spectra: Predicting Cross-lingual Task Performance with Spectral Similarity Measures](https://mlsanity.com/paper/2001.11136/) (2020)

## Abstract (arXiv)

The distributed representations currently used are dense and uninterpretable, leading to interpretations that themselves are relative, overcomplete, and hard to interpret. We propose a method that transforms these word vectors into reduced syntactic representations. The resulting representations are compact and interpretable allowing better visualization and comparison of the word vectors and we successively demonstrate that the drawn interpretations are in line with human judgment. The syntactic representations are then used to create hierarchical word vectors using an incremental learning approach similar to the hierarchical aspect of human learning. As these representations are drawn from pre-trained vectors, the generation process and learning approach are computationally efficient. Most importantly, we find out that syntactic representations provide a plausible interpretation of the vectors and subsequent hierarchical vectors outperform the original vectors in benchmark tests.