# Learning K-way D-dimensional Discrete Codes for Compact Embedding Representations

arXiv: https://arxiv.org/abs/1806.09464 · HTML: https://mlsanity.com/paper/1806.09464/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jun 2018.

## Built on by (most cited first, 11 of 11)

- [Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning](https://mlsanity.com/paper/2208.04202/) (2022)
- [Learning Vector-Quantized Item Representation for Transferable Sequential Recommenders](https://mlsanity.com/paper/2210.12316/) (2022)
- [Lightweight Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/2108.11333/) (2021)
- [Experimental Analysis of Large-scale Learnable Vector Storage Compression](https://mlsanity.com/paper/2311.15578/) (2023)
- [Binary Code based Hash Embedding for Web-scale Applications](https://mlsanity.com/paper/2109.02471/) (2021)
- [Differentiable Product Quantization for Memory Efficient Camera Relocalization](https://mlsanity.com/paper/2407.15540/) (2024)
- [Case-based Reasoning for Better Generalization in Textual Reinforcement Learning](https://mlsanity.com/paper/2110.08470/) (2021)
- [Projected Hamming Dissimilarity for Bit-Level Importance Coding in Collaborative Filtering](https://mlsanity.com/paper/2103.14455/) (2021)
- [Set2Box: Similarity Preserving Representation Learning of Sets](https://mlsanity.com/paper/2210.03282/) (2022)
- [TensorSLM: Energy-efficient Embedding Compression of Sub-billion Parameter Language Models on Low-end Devices](https://mlsanity.com/paper/2506.13514/) (2025)
- [Balanced Co-Clustering of Users and Items for Embedding Table Compression in Recommender Systems](https://mlsanity.com/paper/2604.18351/) (2026)

## Abstract (arXiv)

Conventional embedding methods directly associate each symbol with a continuous embedding vector, which is equivalent to applying a linear transformation based on a "one-hot" encoding of the discrete symbols. Despite its simplicity, such approach yields the number of parameters that grows linearly with the vocabulary size and can lead to overfitting. In this work, we propose a much more compact K-way D-dimensional discrete encoding scheme to replace the "one-hot" encoding. In the proposed "KD encoding", each symbol is represented by a D-dimensional code with a cardinality of K, and the final symbol embedding vector is generated by composing the code embedding vectors. To end-to-end learn semantically meaningful codes, we derive a relaxed discrete optimization approach based on stochastic gradient descent, which can be generally applied to any differentiable computational graph with an embedding layer. In our experiments with various applications from natural language processing to graph convolutional networks, the total size of the embedding layer can be reduced up to 98% while achieving similar or better performance.