# Enhancing Interpretability using Human Similarity Judgements to Prune Word Embeddings

arXiv: https://arxiv.org/abs/2310.10262 · HTML: https://mlsanity.com/paper/2310.10262/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 27 references that are in the map.
- First posted on arXiv 16 Oct 2023.

## Builds on (most cited first, 10 of 10)

- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [A fine-grained comparison of pragmatic language understanding in humans and language models](https://mlsanity.com/paper/2212.06801/) (2022)
- [A Survey on Model Compression and Acceleration for Pretrained Language Models](https://mlsanity.com/paper/2202.07105/) (2022)
- [Problems With Evaluation of Word Embeddings Using Word Similarity Tasks](https://mlsanity.com/paper/1605.02276/) (2016)
- [Training Debiased Subnetworks with Contrastive Weight Pruning](https://mlsanity.com/paper/2210.05247/) (2022)
- [CoSimLex: A Resource for Evaluating Graded Word Similarity in Context](https://mlsanity.com/paper/1912.05320/) (2019)
- [Human Inspired Progressive Alignment and Comparative Learning for Grounded Word Acquisition](https://mlsanity.com/paper/2307.02615/) (2023)

## Built on by (most cited first, 6 of 6)

- [Leveraging a Cognitive Model to Measure Subjective Similarity of Human and GPT-4 Written Content](https://mlsanity.com/paper/2409.00269/) (2024)
- [Improving Interpretability of Lexical Semantic Change with Neurobiological Features](https://mlsanity.com/paper/2602.09760/) (2026)
- [Identifying and interpreting non-aligned human conceptual representations using language modeling](https://mlsanity.com/paper/2403.06204/) (2024)
- [Explaining Human Comparisons using Alignment-Importance Heatmaps](https://mlsanity.com/paper/2409.16292/) (2024)
- [Model Connectomes: A Generational Approach to Data-Efficient Language Models](https://mlsanity.com/paper/2504.21047/) (2025)
- [Grounding Text Embeddings in Stakeholder Associations](https://mlsanity.com/paper/2605.27168/) (2026)

## Abstract (arXiv)

Interpretability methods in NLP aim to provide insights into the semantics underlying specific system architectures. Focusing on word embeddings, we present a supervised-learning method that, for a given domain (e.g., sports, professions), identifies a subset of model features that strongly improve prediction of human similarity judgments. We show this method keeps only 20-40% of the original embeddings, for 8 independent semantic domains, and that it retains different feature sets across domains. We then present two approaches for interpreting the semantics of the retained features. The first obtains the scores of the domain words (co-hyponyms) on the first principal component of the retained embeddings, and extracts terms whose co-occurrence with the co-hyponyms tracks these scores' profile. This analysis reveals that humans differentiate e.g. sports based on how gender-inclusive and international they are. The second approach uses the retained sets as variables in a probing task that predicts values along 65 semantically annotated dimensions for a dataset of 535 words. The features retained for professions are best at predicting cognitive, emotional and social dimensions, whereas features retained for fruits or vegetables best predict the gustation (taste) dimension. We discuss implications for alignment between AI systems and human knowledge.