# Fine-Grained Embedding Dimension Optimization During Training for Recommender Systems

arXiv: https://arxiv.org/abs/2401.04408 · HTML: https://mlsanity.com/paper/2401.04408/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 56 references that are in the map.
- First posted on arXiv 9 Jan 2024.

## Builds on (most cited first, 30 of 30)

- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [Deep Learning Recommendation Model for Personalization and Recommendation Systems](https://mlsanity.com/paper/1906.00091/) (2019)
- [AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks](https://mlsanity.com/paper/1810.11921/) (2018)
- [Beyond Data and Model Parallelism for Deep Neural Networks](https://mlsanity.com/paper/1807.05358/) (2018)
- [Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems](https://mlsanity.com/paper/2003.05622/) (2020)
- [AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations](https://mlsanity.com/paper/2002.11252/) (2020)
- [User Behavior Retrieval for Click-Through Rate Prediction](https://mlsanity.com/paper/2005.14171/) (2020)
- [Learnable Embedding Sizes for Recommender Systems](https://mlsanity.com/paper/2101.07577/) (2021)
- [Mixed Dimension Embeddings with Application to Memory-Efficient Recommendation Systems](https://mlsanity.com/paper/1909.11810/) (2019)
- [Neural Input Search for Large Scale Recommendation Models](https://mlsanity.com/paper/1907.04471/) (2019)
- [Understanding Scaling Laws for Recommendation Models](https://mlsanity.com/paper/2208.08489/) (2022)
- [Model Size Reduction Using Frequency Based Double Hashing for Recommender Systems](https://mlsanity.com/paper/2007.14523/) (2020)
- [AutoML for Deep Recommender Systems: A Survey](https://mlsanity.com/paper/2203.13922/) (2022)
- [HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Framework](https://mlsanity.com/paper/2112.07221/) (2021)
- [RecShard: Statistical Feature-Based Memory Optimization for Industry-Scale Neural Recommendation](https://mlsanity.com/paper/2201.10095/) (2022)
- [Post-Training 4-bit Quantization on Embedding Tables](https://mlsanity.com/paper/1911.02079/) (2019)
- [Merlin HugeCTR: GPU-accelerated Recommender System Training and Inference](https://mlsanity.com/paper/2210.08803/) (2022)
- [Learning Elastic Embeddings for Customizing On-Device Recommenders](https://mlsanity.com/paper/2106.02223/) (2021)
- [Mixed-Precision Embedding Using a Cache](https://mlsanity.com/paper/2010.11305/) (2020)
- [Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameters](https://mlsanity.com/paper/2111.05897/) (2021)
- [Differentiable Neural Input Search for Recommender Systems](https://mlsanity.com/paper/2006.04466/) (2020)
- [Bandana: Using Non-volatile Memory for Storing Deep Learning Models](https://mlsanity.com/paper/1811.05922/) (2018)
- [OptEmbed: Learning Optimal Embedding Table for Click-through Rate Prediction](https://mlsanity.com/paper/2208.04482/) (2022)
- [Learning Compressed Embeddings for On-Device Inference](https://mlsanity.com/paper/2203.10135/) (2022)
- [Single-shot Embedding Dimension Search in Recommender System](https://mlsanity.com/paper/2204.03281/) (2022)
- [Learning Effective and Efficient Embedding via an Adaptively-Masked Twins-based Layer](https://mlsanity.com/paper/2108.11513/) (2021)
- [Binary Code based Hash Embedding for Web-scale Applications](https://mlsanity.com/paper/2109.02471/) (2021)
- [Training with Multi-Layer Embeddings for Model Reduction](https://mlsanity.com/paper/2006.05623/) (2020)
- [A Frequency-aware Software Cache for Large Recommendation System Embeddings](https://mlsanity.com/paper/2208.05321/) (2022)
- [Field-wise Embedding Size Search via Structural Hard Auxiliary Mask Pruning for Click-Through Rate Prediction](https://mlsanity.com/paper/2208.08004/) (2022)

## Built on by (most cited first, 1 of 1)

- [ProactivePIM: Accelerating Weight-Sharing Embedding Layer with PIM for Scalable Recommendation System](https://mlsanity.com/paper/2402.04032/) (2024)

## Abstract (arXiv)

Huge embedding tables in modern deep learning recommender models (DLRM) require prohibitively large memory during training and inference. This paper proposes FIITED, a system to automatically reduce the memory footprint via FIne-grained In-Training Embedding Dimension pruning. By leveraging the key insight that embedding vectors are not equally important, FIITED adaptively adjusts the dimension of each individual embedding vector during model training, assigning larger dimensions to more important embeddings while adapting to dynamic changes in data. We prioritize embedding dimensions with higher frequencies and gradients as more important. To enable efficient pruning of embeddings and their dimensions during model training, we propose an embedding storage system based on virtually-hashed physically-indexed hash tables. Experiments on two industry models and months of realistic datasets show that FIITED can reduce DLRM embedding size by more than 65% while preserving model quality, outperforming state-of-the-art in-training embedding pruning methods. On public datasets, FIITED can reduce the size of embedding tables by 2.1x to 800x with negligible accuracy drop, while improving model throughput.