# Towards Understanding the Overfitting Phenomenon of Deep Click-Through Rate Prediction Models

arXiv: https://arxiv.org/abs/2209.06053 · HTML: https://mlsanity.com/paper/2209.06053/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 35 references that are in the map.
- First posted on arXiv 4 Sep 2022.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Wide & Deep Learning for Recommender Systems](https://mlsanity.com/paper/1606.07792/) (2016)
- [Spectrally-normalized margin bounds for neural networks](https://mlsanity.com/paper/1706.08498/) (2017)
- [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://mlsanity.com/paper/1703.04247/) (2017)
- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [Deep Interest Evolution Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1809.03672/) (2018)
- [Entire Space Multi-Task Model: An Effective Approach for Estimating Post-Click Conversion Rate](https://mlsanity.com/paper/1804.07931/) (2018)
- [One Model to Serve All: Star Topology Adaptive Recommender for Multi-Domain CTR Prediction](https://mlsanity.com/paper/2101.11427/) (2021)
- [Open Benchmarking for Click-Through Rate Prediction](https://mlsanity.com/paper/2009.05794/) (2020)
- [Product-based Neural Networks for User Response Prediction over Multi-field Categorical Data](https://mlsanity.com/paper/1807.00311/) (2018)
- [Overfitting Mechanism and Avoidance in Deep Neural Networks](https://mlsanity.com/paper/1901.06566/) (2019)
- [Real Negatives Matter: Continuous Training with Real Negatives for Delayed Feedback Modeling](https://mlsanity.com/paper/2104.14121/) (2021)
- [PICASSO: Unleashing the Potential of GPU-centric Training for Wide-and-deep Recommender Systems](https://mlsanity.com/paper/2204.04903/) (2022)

## Built on by (most cited first, 30 of 33)

- [Enhancing Taobao Display Advertising with Multimodal Representations: Challenges, Approaches and Insights](https://mlsanity.com/paper/2407.19467/) (2024)
- [AlignRec: Aligning and Training in Multimodal Recommendations](https://mlsanity.com/paper/2403.12384/) (2024)
- [Entire Space Counterfactual Learning for Reliable Content Recommendations](https://mlsanity.com/paper/2210.11039/) (2022)
- [Rec4Ad: A Free Lunch to Mitigate Sample Selection Bias for Ads CTR Prediction in Taobao](https://mlsanity.com/paper/2306.03527/) (2023)
- [Capturing Conversion Rate Fluctuation during Sales Promotions: A Novel Historical Data Reuse Approach](https://mlsanity.com/paper/2305.12837/) (2023)
- [MUSE: A Simple Yet Effective Multimodal Search-Based Framework for Lifelong User Interest Modeling](https://mlsanity.com/paper/2512.07216/) (2025)
- [See Beyond a Single View: Multi-Attribution Learning Leads to Better Conversion Rate Prediction](https://mlsanity.com/paper/2508.15217/) (2025)
- [Multi-Epoch Learning for Deep Click-Through Rate Prediction Models](https://mlsanity.com/paper/2305.19531/) (2023)
- [ESMC: Entire Space Multi-Task Model for Post-Click Conversion Rate via Parameter Constraint](https://mlsanity.com/paper/2307.09193/) (2023)
- [The Evolution of Embedding Table Optimization and Multi-Epoch Training in Pinterest Ads Conversion](https://mlsanity.com/paper/2505.05605/) (2025)
- [COURIER: Contrastive User Intention Reconstruction for Large-Scale Visual Recommendation](https://mlsanity.com/paper/2306.05001/) (2023)
- [Towards Hybrid-grained Feature Interaction Selection for Deep Sparse Network](https://mlsanity.com/paper/2310.15342/) (2023)
- [Data Augmentation in Human-Centric Vision](https://mlsanity.com/paper/2403.08650/) (2024)
- [A Self-boosted Framework for Calibrated Ranking](https://mlsanity.com/paper/2406.08010/) (2024)
- [Multi-Epoch learning with Data Augmentation for Deep Click-Through Rate Prediction](https://mlsanity.com/paper/2407.01607/) (2024)
- [Learning Cascade Ranking as One Network](https://mlsanity.com/paper/2503.09492/) (2025)
- [Taming the One-Epoch Phenomenon in Online Recommendation System by Two-stage Contrastive ID Pre-training](https://mlsanity.com/paper/2508.18700/) (2025)
- [Adaptive Regularization for Large-Scale Sparse Feature Embedding Models](https://mlsanity.com/paper/2511.06374/) (2025)
- [Delayed Feedback Modeling for Post-Click Gross Merchandise Volume Prediction: Benchmark, Insights and Approaches](https://mlsanity.com/paper/2601.20307/) (2026)
- [IDProxy: CTR Prediction with Multimodal LLMs for Cold-Start Recommendation at Xiaohongshu](https://mlsanity.com/paper/2603.01590/) (2026)
- [Mobile Supply: The Last Piece of Jigsaw of Recommender System](https://mlsanity.com/paper/2308.03855/) (2023)
- [Interpretable Triplet Importance for Personalized Ranking](https://mlsanity.com/paper/2407.19469/) (2024)
- [Multi-Faceted Large Embedding Tables for Pinterest Ads Ranking](https://mlsanity.com/paper/2508.05700/) (2025)
- [RQ-GMM: Residual Quantized Gaussian Mixture Model for Multimodal Semantic Discretization in CTR Prediction](https://mlsanity.com/paper/2602.12593/) (2026)
- [MAC: A Conversion Rate Prediction Benchmark Featuring Labels Under Multiple Attribution Mechanisms](https://mlsanity.com/paper/2603.02184/) (2026)
- [EMOFM: Ensemble MLP mOdel with Feature-based Mixers for Click-Through Rate Prediction](https://mlsanity.com/paper/2310.04482/) (2023)
- [UMAIR-FPS: User-aware Multi-modal Animation Illustration Recommendation Fusion with Painting Style](https://mlsanity.com/paper/2402.10381/) (2024)
- [PRECTR: A Synergistic Framework for Integrating Personalized Search Relevance Matching and CTR Prediction](https://mlsanity.com/paper/2503.18395/) (2025)
- [A Probabilistic Framework for Temporal Distribution Generalization in Industry-Scale Recommender Systems](https://mlsanity.com/paper/2511.21032/) (2025)
- [Distribution-Aware End-to-End Embedding for Streaming Numerical Features in Click-Through Rate Prediction](https://mlsanity.com/paper/2602.03223/) (2026)

## Abstract (arXiv)

Deep learning techniques have been applied widely in industrial recommendation systems. However, far less attention has been paid to the overfitting problem of models in recommendation systems, which, on the contrary, is recognized as a critical issue for deep neural networks. In the context of Click-Through Rate (CTR) prediction, we observe an interesting one-epoch overfitting problem: the model performance exhibits a dramatic degradation at the beginning of the second epoch. Such a phenomenon has been witnessed widely in real-world applications of CTR models. Thereby, the best performance is usually achieved by training with only one epoch. To understand the underlying factors behind the one-epoch phenomenon, we conduct extensive experiments on the production data set collected from the display advertising system of Alibaba. The results show that the model structure, the optimization algorithm with a fast convergence rate, and the feature sparsity are closely related to the one-epoch phenomenon. We also provide a likely hypothesis for explaining such a phenomenon and conduct a set of proof-of-concept experiments. We hope this work can shed light on future research on training more epochs for better performance.