# Open Benchmarking for Click-Through Rate Prediction

arXiv: https://arxiv.org/abs/2009.05794 · HTML: https://mlsanity.com/paper/2009.05794/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Sep 2020.

## Built on by (most cited first, 30 of 53)

- [Wukong: Towards a Scaling Law for Large-Scale Recommendation](https://mlsanity.com/paper/2403.02545/) (2024)
- [EasyTPP: Towards Open Benchmarking Temporal Point Processes](https://mlsanity.com/paper/2307.08097/) (2023)
- [Towards Understanding the Overfitting Phenomenon of Deep Click-Through Rate Prediction Models](https://mlsanity.com/paper/2209.06053/) (2022)
- [Joint Optimization of Ranking and Calibration with Contextualized Hybrid Model](https://mlsanity.com/paper/2208.06164/) (2022)
- [STEM: Unleashing the Power of Embeddings for Multi-task Recommendation](https://mlsanity.com/paper/2308.13537/) (2023)
- [Beyond Two-Tower Matching: Learning Sparse Retrievable Cross-Interactions for Recommendation](https://mlsanity.com/paper/2311.18213/) (2023)
- [External Large Foundation Model: How to Efficiently Serve Trillions of Parameters for Online Ads Recommendation](https://mlsanity.com/paper/2502.17494/) (2025)
- [PEAR: Personalized Re-ranking with Contextualized Transformer for Recommendation](https://mlsanity.com/paper/2203.12267/) (2022)
- [Experimental Analysis of Large-scale Learnable Vector Storage Compression](https://mlsanity.com/paper/2311.15578/) (2023)
- [EB-NeRD: A Large-Scale Dataset for News Recommendation](https://mlsanity.com/paper/2410.03432/) (2024)
- [Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders](https://mlsanity.com/paper/2507.16289/) (2025)
- [Adaptive Low-Precision Training for Embeddings in Click-Through Rate Prediction](https://mlsanity.com/paper/2212.05735/) (2022)
- [Capturing Conversion Rate Fluctuation during Sales Promotions: A Novel Historical Data Reuse Approach](https://mlsanity.com/paper/2305.12837/) (2023)
- [CAFE: Towards Compact, Adaptive, and Fast Embedding for Large-scale Recommendation Models](https://mlsanity.com/paper/2312.03256/) (2023)
- [MerRec: A Large-scale Multipurpose Mercari Dataset for Consumer-to-Consumer Recommendation Systems](https://mlsanity.com/paper/2402.14230/) (2024)
- [FairJob: A Real-World Dataset for Fairness in Online Systems](https://mlsanity.com/paper/2407.03059/) (2024)
- [A Comprehensive Survey on Automated Machine Learning for Recommendations](https://mlsanity.com/paper/2204.01390/) (2022)
- [CowClip: Reducing CTR Prediction Model Training Time from 12 hours to 10 minutes on 1 GPU](https://mlsanity.com/paper/2204.06240/) (2022)
- [AGD: an Auto-switchable Optimizer using Stepwise Gradient Difference for Preconditioning Matrix](https://mlsanity.com/paper/2312.01658/) (2023)
- [A Thorough Performance Benchmarking on Lightweight Embedding-based Recommender Systems](https://mlsanity.com/paper/2406.17335/) (2024)
- [Field Matters: A Lightweight LLM-enhanced Method for CTR Prediction](https://mlsanity.com/paper/2505.14057/) (2025)
- [Exploring Recommender System Evaluation: A Multi-Modal User Agent Framework for A/B Testing](https://mlsanity.com/paper/2601.04554/) (2026)
- [PACEvolve++: Improving Test-time Learning for Evolutionary Search Agents](https://mlsanity.com/paper/2605.07039/) (2026)
- [EasyRec: An easy-to-use, extendable and efficient framework for building industrial recommendation systems](https://mlsanity.com/paper/2209.12766/) (2022)
- [Benchmarking for Deep Uplift Modeling in Online Marketing](https://mlsanity.com/paper/2406.00335/) (2024)
- [Revisiting Feature Interactions from the Perspective of Quadratic Neural Networks for Click-through Rate Prediction](https://mlsanity.com/paper/2505.17999/) (2025)
- [ReasonRec: A Reasoning-Augmented Multimodal Agent for Unified Recommendation](https://mlsanity.com/paper/2606.28357/) (2026)
- [Towards Hybrid-grained Feature Interaction Selection for Deep Sparse Network](https://mlsanity.com/paper/2310.15342/) (2023)
- [CAdam: Confidence-Based Optimization for Online Learning](https://mlsanity.com/paper/2411.19647/) (2024)
- [Macro Graph of Experts for Billion-Scale Multi-Task Recommendation](https://mlsanity.com/paper/2506.10520/) (2025)

## Abstract (arXiv)

Click-through rate (CTR) prediction is a critical task for many applications, as its accuracy has a direct impact on user experience and platform revenue. In recent years, CTR prediction has been widely studied in both academia and industry, resulting in a wide variety of CTR prediction models. Unfortunately, there is still a lack of standardized benchmarks and uniform evaluation protocols for CTR prediction research. This leads to non-reproducible or even inconsistent experimental results among existing studies, which largely limits the practical value and potential impact of their research. In this work, we build an open benchmark for CTR prediction, namely BARS-CTR, and present a rigorous comparison of different models in a reproducible manner. To this end, we ran over 7,000 experiments for more than 12,000 GPU hours in total to re-evaluate 24 existing models on multiple datasets and settings. Surprisingly, our experiments show that with sufficient hyper-parameter search and model tuning, many deep models have smaller differences than expected. The results also reveal that making real progress on the modeling of CTR prediction is indeed a very challenging research task. We believe that our benchmarking work could not only allow researchers to gauge the effectiveness of new models conveniently but also make them fairly compare with the state of the arts. We have publicly released the benchmarking code, evaluation protocols, and hyper-parameter settings of our work to promote reproducible research in this field.