# Near-lossless Binarization of Word Embeddings

arXiv: https://arxiv.org/abs/1803.09065 · HTML: https://mlsanity.com/paper/1803.09065/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Mar 2018.

## Built on by (most cited first, 13 of 13)

- [Can a Fruit Fly Learn Word Embeddings?](https://mlsanity.com/paper/2101.06887/) (2021)
- [The Rediscovery Hypothesis: Language Models Need to Meet Linguistics](https://mlsanity.com/paper/2103.01819/) (2021)
- [Ultra-High Dimensional Sparse Representations with Binarization for Efficient Text Retrieval](https://mlsanity.com/paper/2104.07198/) (2021)
- [MorphTE: Injecting Morphology in Tensorized Embeddings](https://mlsanity.com/paper/2210.15379/) (2022)
- [BTR: Binary Token Representations for Efficient Retrieval Augmented Language Models](https://mlsanity.com/paper/2310.01329/) (2023)
- [Optimal Size-Performance Tradeoffs: Weighing PoS Tagger Models](https://mlsanity.com/paper/2104.07951/) (2021)
- [From Fully Trained to Fully Random Embeddings: Improving Neural Machine Translation with Compact Word Embedding Tables](https://mlsanity.com/paper/2104.08677/) (2021)
- [Word Tour: One-dimensional Word Embeddings via the Traveling Salesman Problem](https://mlsanity.com/paper/2205.01954/) (2022)
- [Enhancing Cross-Category Learning in Recommendation Systems with Multi-Layer Embedding Training](https://mlsanity.com/paper/2309.15881/) (2023)
- [SNNLP: Energy-Efficient Natural Language Processing Using Spiking Neural Networks](https://mlsanity.com/paper/2401.17911/) (2024)
- [Evolutionary Feature-wise Thresholding for Binary Representation of NLP Embeddings](https://mlsanity.com/paper/2507.17025/) (2025)
- [Word2Spike: Poisson Rate Coding for Associative Memories and Neuromorphic Algorithms](https://mlsanity.com/paper/2509.07361/) (2025)
- [TextEconomizer: Enhancing Lossy Text Compression with Denoising Transformers and Entropy Coding](https://mlsanity.com/paper/2606.08184/) (2026)

## Abstract (arXiv)

Word embeddings are commonly used as a starting point in many NLP models to achieve state-of-the-art performances. However, with a large vocabulary and many dimensions, these floating-point representations are expensive both in terms of memory and calculations which makes them unsuitable for use on low-resource devices. The method proposed in this paper transforms real-valued embeddings into binary embeddings while preserving semantic information, requiring only 128 or 256 bits for each vector. This leads to a small memory footprint and fast vector operations. The model is based on an autoencoder architecture, which also allows to reconstruct original vectors from the binary ones. Experimental results on semantic similarity, text classification and sentiment analysis tasks show that the binarization of word embeddings only leads to a loss of ~2% in accuracy while vector size is reduced by 97%. Furthermore, a top-k benchmark demonstrates that using these binary vectors is 30 times faster than using real-valued vectors.