# Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems

arXiv: https://arxiv.org/abs/1909.02107 · HTML: https://mlsanity.com/paper/1909.02107/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Sep 2019.

## Built on by (most cited first, 30 of 68)

- [Learning Vector-Quantized Item Representation for Transferable Sequential Recommenders](https://mlsanity.com/paper/2210.12316/) (2022)
- [PQCache: Product Quantization-based KVCache for Long Context LLM Inference](https://mlsanity.com/paper/2407.12820/) (2024)
- [Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models](https://mlsanity.com/paper/2601.07372/) (2026)
- [A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scale](https://mlsanity.com/paper/2309.06497/) (2023)
- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [Learnable Embedding Sizes for Recommender Systems](https://mlsanity.com/paper/2101.07577/) (2021)
- [A Comprehensive Survey on Trustworthy Recommender Systems](https://mlsanity.com/paper/2209.10117/) (2022)
- [Click-Through Rate Prediction in Online Advertising: A Literature Review](https://mlsanity.com/paper/2202.10462/) (2022)
- [Accelerating Recommendation System Training by Leveraging Popular Choices](https://mlsanity.com/paper/2103.00686/) (2021)
- [Lightweight Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/2108.11333/) (2021)
- [Embedding in Recommender Systems: A Survey](https://mlsanity.com/paper/2310.18608/) (2023)
- [Towards Fair Federated Recommendation Learning: Characterizing the Inter-Dependence of System and Data Heterogeneity](https://mlsanity.com/paper/2206.02633/) (2022)
- [AutoShard: Automated Embedding Table Sharding for Recommender Systems](https://mlsanity.com/paper/2208.06399/) (2022)
- [DreamShard: Generalizable Embedding Table Placement for Recommender Systems](https://mlsanity.com/paper/2210.02023/) (2022)
- [Towards energy-efficient Deep Learning: An overview of energy-efficient approaches along the Deep Learning Lifecycle](https://mlsanity.com/paper/2303.01980/) (2023)
- [Unified Embedding: Battle-Tested Feature Representations for Web-Scale ML Systems](https://mlsanity.com/paper/2305.12102/) (2023)
- [Learning Compressed Embeddings for On-Device Inference](https://mlsanity.com/paper/2203.10135/) (2022)
- [Heterogeneous Acceleration Pipeline for Recommendation System Training](https://mlsanity.com/paper/2204.05436/) (2022)
- [Learning Effective and Efficient Embedding via an Adaptively-Masked Twins-based Layer](https://mlsanity.com/paper/2108.11513/) (2021)
- [Experimental Analysis of Large-scale Learnable Vector Storage Compression](https://mlsanity.com/paper/2311.15578/) (2023)
- [LiRank: Industrial Large Scale Ranking Models at LinkedIn](https://mlsanity.com/paper/2402.06859/) (2024)
- [Binary Code based Hash Embedding for Web-scale Applications](https://mlsanity.com/paper/2109.02471/) (2021)
- [Adaptive Low-Precision Training for Embeddings in Click-Through Rate Prediction](https://mlsanity.com/paper/2212.05735/) (2022)
- [MP-Rec: Hardware-Software Co-Design to Enable Multi-Path Recommendation](https://mlsanity.com/paper/2302.10872/) (2023)
- [RecPipe: Co-designing Models and Hardware to Jointly Optimize Recommendation Quality and Performance](https://mlsanity.com/paper/2105.08820/) (2021)
- [Semantically Constrained Memory Allocation (SCMA) for Embedding in Efficient Recommendation Systems](https://mlsanity.com/paper/2103.06124/) (2021)
- [CAFE: Towards Compact, Adaptive, and Fast Embedding for Large-scale Recommendation Models](https://mlsanity.com/paper/2312.03256/) (2023)
- [Differentiable NAS Framework and Application to Ads CTR Prediction](https://mlsanity.com/paper/2110.14812/) (2021)
- [Nimble GNN Embedding with Tensor-Train Decomposition](https://mlsanity.com/paper/2206.10581/) (2022)
- [Learning to Collide: Recommendation System Model Compression with Learned Hash Functions](https://mlsanity.com/paper/2203.15837/) (2022)

## Abstract (arXiv)

Modern deep learning-based recommendation systems exploit hundreds to thousands of different categorical features, each with millions of different categories ranging from clicks to posts. To respect the natural diversity within the categorical data, embeddings map each category to a unique dense representation within an embedded space. Since each categorical feature could take on as many as tens of millions of different possible categories, the embedding tables form the primary memory bottleneck during both training and inference. We propose a novel approach for reducing the embedding size in an end-to-end fashion by exploiting complementary partitions of the category set to produce a unique embedding vector for each category without explicit definition. By storing multiple smaller embedding tables based on each complementary partition and combining embeddings from each table, we define a unique embedding for each category at smaller memory cost. This approach may be interpreted as using a specific fixed codebook to ensure uniqueness of each category's representation. Our experimental results demonstrate the effectiveness of our approach over the hashing trick for reducing the size of the embedding tables in terms of model loss and accuracy, while retaining a similar reduction in the number of parameters.