# Adaptive Low-Precision Training for Embeddings in Click-Through Rate Prediction

arXiv: https://arxiv.org/abs/2212.05735 · HTML: https://mlsanity.com/paper/2212.05735/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 39 references that are in the map.
- First posted on arXiv 12 Dec 2022.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [Wide & Deep Learning for Recommender Systems](https://mlsanity.com/paper/1606.07792/) (2016)
- [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://mlsanity.com/paper/1703.04247/) (2017)
- [Learned Step Size Quantization](https://mlsanity.com/paper/1902.08153/) (2019)
- [Up or Down? Adaptive Rounding for Post-Training Quantization](https://mlsanity.com/paper/2004.10568/) (2020)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [PACT: Parameterized Clipping Activation for Quantized Neural Networks](https://mlsanity.com/paper/1805.06085/) (2018)
- [Deep & Cross Network for Ad Click Predictions](https://mlsanity.com/paper/1708.05123/) (2017)
- [Training Quantized Nets: A Deeper Understanding](https://mlsanity.com/paper/1706.02379/) (2017)
- [Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems](https://mlsanity.com/paper/1909.02107/) (2019)
- [TT-Rec: Tensor Train Compression for Deep Learning Recommendation Models](https://mlsanity.com/paper/2101.11714/) (2021)
- [Loss-aware Binarization of Deep Networks](https://mlsanity.com/paper/1611.01600/) (2016)
- [Open Benchmarking for Click-Through Rate Prediction](https://mlsanity.com/paper/2009.05794/) (2020)
- [Neural Networks with Few Multiplications](https://mlsanity.com/paper/1510.03009/) (2015)
- [Learnable Embedding Sizes for Recommender Systems](https://mlsanity.com/paper/2101.07577/) (2021)
- [Neural Input Search for Large Scale Recommendation Models](https://mlsanity.com/paper/1907.04471/) (2019)
- [An Embedding Learning Framework for Numerical Features in CTR Prediction](https://mlsanity.com/paper/2012.08986/) (2020)
- [Model Size Reduction Using Frequency Based Double Hashing for Recommender Systems](https://mlsanity.com/paper/2007.14523/) (2020)
- [Loss-aware Weight Quantization of Deep Networks](https://mlsanity.com/paper/1802.08635/) (2018)
- [Post-Training 4-bit Quantization on Embedding Tables](https://mlsanity.com/paper/1911.02079/) (2019)
- [Detecting Beneficial Feature Interactions for Recommender Systems](https://mlsanity.com/paper/2008.00404/) (2020)
- [On-Device Next-Item Recommendation with Self-Supervised Knowledge Distillation](https://mlsanity.com/paper/2204.11091/) (2022)
- [Learning Elastic Embeddings for Customizing On-Device Recommenders](https://mlsanity.com/paper/2106.02223/) (2021)
- [Mixed-Precision Embedding Using a Cache](https://mlsanity.com/paper/2010.11305/) (2020)
- [Memory-efficient Embedding for Recommendations](https://mlsanity.com/paper/2006.14827/) (2020)
- [OptEmbed: Learning Optimal Embedding Table for Click-through Rate Prediction](https://mlsanity.com/paper/2208.04482/) (2022)
- [ScaleFreeCTR: MixCache-based Distributed Training System for CTR Models with Huge Embedding Table](https://mlsanity.com/paper/2104.08542/) (2021)

## Built on by (most cited first, 11 of 11)

- [Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics](https://mlsanity.com/paper/2505.23194/) (2025)
- [CAFE: Towards Compact, Adaptive, and Fast Embedding for Large-scale Recommendation Models](https://mlsanity.com/paper/2312.03256/) (2023)
- [Semantic Retrieval Augmented Contrastive Learning for Sequential Recommendation](https://mlsanity.com/paper/2503.04162/) (2025)
- [Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation](https://mlsanity.com/paper/2510.23123/) (2025)
- [The Panaceas for Improving Low-Rank Decomposition in Communication-Efficient Federated Learning](https://mlsanity.com/paper/2505.23176/) (2025)
- [A Thorough Performance Benchmarking on Lightweight Embedding-based Recommender Systems](https://mlsanity.com/paper/2406.17335/) (2024)
- [The Future is Sparse: Embedding Compression for Scalable Retrieval in Recommender Systems](https://mlsanity.com/paper/2505.11388/) (2025)
- [FedBiF: Communication-Efficient Federated Learning via Bits Freezing](https://mlsanity.com/paper/2509.10161/) (2025)
- [ProactivePIM: Accelerating Weight-Sharing Embedding Layer with PIM for Scalable Recommendation System](https://mlsanity.com/paper/2402.04032/) (2024)
- [ShuffleGate: Scalable Feature Optimization for Recommender Systems via Batch-wise Sensitivity Learning](https://mlsanity.com/paper/2503.09315/) (2025)
- [DimGrow: Memory-Efficient Field-level Embedding Dimension Search](https://mlsanity.com/paper/2505.12683/) (2025)

## Abstract (arXiv)

Embedding tables are usually huge in click-through rate (CTR) prediction models. To train and deploy the CTR models efficiently and economically, it is necessary to compress their embedding tables at the training stage. To this end, we formulate a novel quantization training paradigm to compress the embeddings from the training stage, termed low-precision training (LPT). Also, we provide theoretical analysis on its convergence. The results show that stochastic weight quantization has a faster convergence rate and a smaller convergence error than deterministic weight quantization in LPT. Further, to reduce the accuracy degradation, we propose adaptive low-precision training (ALPT) that learns the step size (i.e., the quantization resolution) through gradient descent. Experiments on two real-world datasets confirm our analysis and show that ALPT can significantly improve the prediction accuracy, especially at extremely low bit widths. For the first time in CTR models, we successfully train 8-bit embeddings without sacrificing prediction accuracy. The code of ALPT is publicly available.