# Compressing Word Embeddings via Deep Compositional Code Learning

arXiv: https://arxiv.org/abs/1711.01068 · HTML: https://mlsanity.com/paper/1711.01068/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Nov 2017.

## Built on by (most cited first, 30 of 35)

- [A Review of the Gumbel-max Trick and its Extensions for Discrete Stochasticity in Machine Learning](https://mlsanity.com/paper/2110.01515/) (2021)
- [Federated Learning of Gboard Language Models with Differential Privacy](https://mlsanity.com/paper/2305.18465/) (2023)
- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [Lightweight Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/2108.11333/) (2021)
- [Towards energy-efficient Deep Learning: An overview of energy-efficient approaches along the Deep Learning Lifecycle](https://mlsanity.com/paper/2303.01980/) (2023)
- [Experimental Analysis of Large-scale Learnable Vector Storage Compression](https://mlsanity.com/paper/2311.15578/) (2023)
- [Binary Code based Hash Embedding for Web-scale Applications](https://mlsanity.com/paper/2109.02471/) (2021)
- [Efficient Inference for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2109.06679/) (2021)
- [Embedding Compression with Hashing for Efficient Representation Learning in Large-Scale Graph](https://mlsanity.com/paper/2208.05648/) (2022)
- [Semi-supervised Network Embedding with Differentiable Deep Quantisation](https://mlsanity.com/paper/2108.09128/) (2021)
- [HashFormers: Towards Vocabulary-independent Pre-trained Transformers](https://mlsanity.com/paper/2210.07904/) (2022)
- [Evaluating Unsupervised Dimensionality Reduction Methods for Pretrained Sentence Embeddings](https://mlsanity.com/paper/2403.14001/) (2024)
- [MorphTE: Injecting Morphology in Tensorized Embeddings](https://mlsanity.com/paper/2210.15379/) (2022)
- [Projected Hamming Dissimilarity for Bit-Level Importance Coding in Collaborative Filtering](https://mlsanity.com/paper/2103.14455/) (2021)
- [Composite Code Sparse Autoencoders for first stage retrieval](https://mlsanity.com/paper/2204.07023/) (2022)
- [Extending Multi-Sense Word Embedding to Phrases and Sentences for Unsupervised Semantic Applications](https://mlsanity.com/paper/2103.15330/) (2021)
- [From Fully Trained to Fully Random Embeddings: Improving Neural Machine Translation with Compact Word Embedding Tables](https://mlsanity.com/paper/2104.08677/) (2021)
- [Learning Compositional Shape Priors for Few-Shot 3D Reconstruction](https://mlsanity.com/paper/2106.06440/) (2021)
- [Gaussian Mixture Vector Quantization with Aggregated Categorical Posterior](https://mlsanity.com/paper/2410.10180/) (2024)
- [Pixel Embedding: Fully Quantized Convolutional Neural Network with Differentiable Lookup Table](https://mlsanity.com/paper/2407.16174/) (2024)
- [MultiTok: Variable-Length Tokenization for Efficient LLMs Adapted from LZW Compression](https://mlsanity.com/paper/2410.21548/) (2024)
- [MultiHashFormer: Hash-based Generative Language Models](https://mlsanity.com/paper/2606.28057/) (2026)
- [Unsupervised Domain-adaptive Hash for Networks](https://mlsanity.com/paper/2108.09136/) (2021)
- [Toward Compact Parameter Representations for Architecture-Agnostic Neural Network Compression](https://mlsanity.com/paper/2111.10320/) (2021)
- [Embedding Compression for Text Classification Using Dictionary Screening](https://mlsanity.com/paper/2211.12715/) (2022)
- [Rediscovering Hashed Random Projections for Efficient Quantization of Contextualized Sentence Embeddings](https://mlsanity.com/paper/2304.02481/) (2023)
- [Word Embedding Dimension Reduction via Weakly-Supervised Feature Selection](https://mlsanity.com/paper/2407.12342/) (2024)
- [Parameter-Efficient Transformer Embeddings](https://mlsanity.com/paper/2505.02266/) (2025)
- [Memory-Efficient FastText: A Comprehensive Approach Using Double-Array Trie Structures and Mark-Compact Memory Management](https://mlsanity.com/paper/2506.01254/) (2025)
- [Semantic Compression for Word and Sentence Embeddings using Discrete Wavelet Transform](https://mlsanity.com/paper/2508.00220/) (2025)

## Abstract (arXiv)

Natural language processing (NLP) models often require a massive number of parameters for word embeddings, resulting in a large storage or memory footprint. Deploying neural NLP models to mobile devices requires compressing the word embeddings without any significant sacrifices in performance. For this purpose, we propose to construct the embeddings with few basis vectors. For each word, the composition of basis vectors is determined by a hash code. To maximize the compression rate, we adopt the multi-codebook quantization approach instead of binary coding scheme. Each code is composed of multiple discrete numbers, such as (3, 2, 1, 8), where the value of each component is limited to a fixed range. We propose to directly learn the discrete codes in an end-to-end neural network by applying the Gumbel-softmax trick. Experiments show the compression rate achieves 98% in a sentiment analysis task and 94% ~ 99% in machine translation tasks without performance loss. In both tasks, the proposed method can improve the model performance by slightly lowering the compression rate. Compared to other approaches such as character-level segmentation, the proposed method is language-independent and does not require modifications to the network architecture.