# Adaptive Regularization for Large-Scale Sparse Feature Embedding Models

arXiv: https://arxiv.org/abs/2511.06374 · HTML: https://mlsanity.com/paper/2511.06374/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 39 references that are in the map.
- First posted on arXiv 9 Nov 2025.

## Builds on (most cited first, 20 of 20)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [Wide & Deep Learning for Recommender Systems](https://mlsanity.com/paper/1606.07792/) (2016)
- [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://mlsanity.com/paper/1703.04247/) (2017)
- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [Size-Independent Sample Complexity of Neural Networks](https://mlsanity.com/paper/1712.06541/) (2017)
- [xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems](https://mlsanity.com/paper/1803.05170/) (2018)
- [Better Theory for SGD in the Nonconvex World](https://mlsanity.com/paper/2002.03329/) (2020)
- [Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmark](https://mlsanity.com/paper/2202.06767/) (2022)
- [Wukong: Towards a Scaling Law for Large-Scale Recommendation](https://mlsanity.com/paper/2403.02545/) (2024)
- [BARS: Towards Open Benchmarking for Recommender Systems](https://mlsanity.com/paper/2205.09626/) (2022)
- [Towards Understanding the Overfitting Phenomenon of Deep Click-Through Rate Prediction Models](https://mlsanity.com/paper/2209.06053/) (2022)
- [On the Embedding Collapse when Scaling up Recommendation Models](https://mlsanity.com/paper/2310.04400/) (2023)
- [PICASSO: Unleashing the Potential of GPU-centric Training for Wide-and-deep Recommender Systems](https://mlsanity.com/paper/2204.04903/) (2022)
- [Convergence of Adam for Non-convex Objectives: Relaxed Hyperparameters and Non-ergodic Case](https://mlsanity.com/paper/2307.11782/) (2023)
- [Scaling Transformers for Discriminative Recommendation via Generative Pretraining](https://mlsanity.com/paper/2506.03699/) (2025)
- [Multi-Epoch Learning for Deep Click-Through Rate Prediction Models](https://mlsanity.com/paper/2305.19531/) (2023)
- [Multi-Epoch learning with Data Augmentation for Deep Click-Through Rate Prediction](https://mlsanity.com/paper/2407.01607/) (2024)

## Built on by (most cited first, 2 of 2)

- [Cross-Country Code-Mixing for Generative Recommendation](https://mlsanity.com/paper/2609.28972/) (2026)
- [Beyond One Epoch: Uncertainty-Weighted Sensitivity Regularization for Recommendation Models](https://mlsanity.com/paper/2609.34083/) (2026)

## Abstract (arXiv)

The one-epoch overfitting problem has drawn widespread attention, especially in CTR and CVR estimation models in search, advertising, and recommendation domains. These models which rely heavily on large-scale sparse categorical features, often suffer a significant decline in performance when trained for multiple epochs. Although recent studies have proposed heuristic solutions, the fundamental cause of this phenomenon remains unclear. In this work, we present a theoretical explanation grounded in Rademacher complexity, supported by empirical experiments, to explain why overfitting occurs in models with large-scale sparse categorical features. Based on this analysis, we propose a regularization method that constrains the norm budget of embedding layers adaptively. Our approach not only prevents the severe performance degradation observed during multi-epoch training, but also improves model performance within a single epoch. This method has already been deployed in online production systems.