# Compressing Word Embeddings

arXiv: https://arxiv.org/abs/1511.06397 · HTML: https://mlsanity.com/paper/1511.06397/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Nov 2015.

## Built on by (most cited first, 3 of 3)

- [Neural Semantic Parsing with Extremely Rich Symbolic Meaning Representations](https://mlsanity.com/paper/2404.12698/) (2024)
- [On Geodesic Distances and Contextual Embedding Compression for Text Classification](https://mlsanity.com/paper/2104.11295/) (2021)
- [Exploring In-Context Learning for Handwritten Text Recognition](https://mlsanity.com/paper/2609.37195/) (2026)

## Abstract (arXiv)

Recent methods for learning vector space representations of words have succeeded in capturing fine-grained semantic and syntactic regularities using vector arithmetic. However, these vector space representations (created through large-scale text analysis) are typically stored verbatim, since their internal structure is opaque. Using word-analogy tests to monitor the level of detail stored in compressed re-representations of the same vector space, the trade-offs between the reduction in memory usage and expressiveness are investigated. A simple scheme is outlined that can reduce the memory footprint of a state-of-the-art embedding by a factor of 10, with only minimal impact on performance. Then, using the same `bit budget', a binary (approximate) factorisation of the same space is also explored, with the aim of creating an equivalent representation with better interpretability.