# Embedding Compression with Isotropic Iterative Quantization

arXiv: https://arxiv.org/abs/2001.05314 · HTML: https://mlsanity.com/paper/2001.05314/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jan 2020.

## Built on by (most cited first, 10 of 10)

- [Stable Anisotropic Regularization](https://mlsanity.com/paper/2305.19358/) (2023)
- [Reliable Measures of Spread in High Dimensional Latent Spaces](https://mlsanity.com/paper/2212.08172/) (2022)
- [Optimal Size-Performance Tradeoffs: Weighing PoS Tagger Models](https://mlsanity.com/paper/2104.07951/) (2021)
- [Exploring the Impact of a Transformer's Latent Space Geometry on Downstream Task Performance](https://mlsanity.com/paper/2406.12159/) (2024)
- [When Is 0.1% Enough? Analyzing the Combined Effects of Dimensionality Reduction and Quantization on Text Embedding Compression](https://mlsanity.com/paper/2606.01074/) (2026)
- [Rediscovering Hashed Random Projections for Efficient Quantization of Contextualized Sentence Embeddings](https://mlsanity.com/paper/2304.02481/) (2023)
- [Extracting Text Representations for Terms and Phrases in Technical Domains](https://mlsanity.com/paper/2305.15867/) (2023)
- [Review of compressed embedding layers and their applications for recommender systems](https://mlsanity.com/paper/2306.13724/) (2023)
- [Vanishing Contributions: A Unified Framework for Smooth and Iterative Model Compression](https://mlsanity.com/paper/2510.09696/) (2025)
- [Reinforcement-Learned Unequal Error Protection for Quantized Semantic Embeddings](https://mlsanity.com/paper/2601.00186/) (2026)

## Abstract (arXiv)

Continuous representation of words is a standard component in deep learning-based NLP models. However, representing a large vocabulary requires significant memory, which can cause problems, particularly on resource-constrained platforms. Therefore, in this paper we propose an isotropic iterative quantization (IIQ) approach for compressing embedding vectors into binary ones, leveraging the iterative quantization technique well established for image retrieval, while satisfying the desired isotropic property of PMI based models. Experiments with pre-trained embeddings (i.e., GloVe and HDC) demonstrate a more than thirty-fold compression ratio with comparable and sometimes even improved performance over the original real-valued embedding vectors.