# Ensemble Knowledge Distillation for CTR Prediction

arXiv: https://arxiv.org/abs/2011.04106 · HTML: https://mlsanity.com/paper/2011.04106/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Nov 2020.

## Built on by (most cited first, 18 of 18)

- [A Comprehensive Survey on Trustworthy Recommender Systems](https://mlsanity.com/paper/2209.10117/) (2022)
- [Click-Through Rate Prediction in Online Advertising: A Literature Review](https://mlsanity.com/paper/2202.10462/) (2022)
- [Teacher-Student Architecture for Knowledge Distillation: A Survey](https://mlsanity.com/paper/2308.04268/) (2023)
- [Topology Distillation for Recommender System](https://mlsanity.com/paper/2106.08700/) (2021)
- [Cross-Task Knowledge Distillation in Multi-Task Recommendation](https://mlsanity.com/paper/2202.09852/) (2022)
- [Distillation from Heterogeneous Models for Top-K Recommendation](https://mlsanity.com/paper/2303.01130/) (2023)
- [External Large Foundation Model: How to Efficiently Serve Trillions of Parameters for Online Ads Recommendation](https://mlsanity.com/paper/2502.17494/) (2025)
- [Unbiased Knowledge Distillation for Recommendation](https://mlsanity.com/paper/2211.14729/) (2022)
- [Learning Task-Agnostic Representations through Multi-Teacher Distillation](https://mlsanity.com/paper/2510.18680/) (2025)
- [Learning-To-Ensemble by Contextual Rank Aggregation in E-Commerce](https://mlsanity.com/paper/2107.08598/) (2021)
- [Directed Acyclic Graph Factorization Machines for CTR Prediction via Knowledge Distillation](https://mlsanity.com/paper/2211.11159/) (2022)
- [Pairwise Ranking Losses of Click-Through Rates Prediction for Welfare Maximization in Ad Auctions](https://mlsanity.com/paper/2306.01799/) (2023)
- [Retrieval and Distill: A Temporal Data Shift-Free Paradigm for Online Recommendation System](https://mlsanity.com/paper/2404.15678/) (2024)
- [FeatureBox: Feature Engineering on GPUs for Massive-Scale Ads Systems](https://mlsanity.com/paper/2210.07768/) (2022)
- [DIIT: A Domain-Invariant Information Transfer Method for Industrial Cross-Domain Recommendation](https://mlsanity.com/paper/2410.10835/) (2024)
- [A Unified Knowledge-Distillation and Semi-Supervised Learning Framework to Improve Industrial Ads Delivery Systems](https://mlsanity.com/paper/2502.06834/) (2025)
- [QueryFormer: Winning Solution for KDD Cup 2026 Tencent UniRec Challenge](https://mlsanity.com/paper/2609.16548/) (2026)
- [Decoupled and Distilled: Task-Adaptive LoRA-Teachers with Ensemble Knowledge Transfer for Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2609.39390/) (2026)

## Abstract (arXiv)

Recently, deep learning-based models have been widely studied for click-through rate (CTR) prediction and lead to improved prediction accuracy in many industrial applications. However, current research focuses primarily on building complex network architectures to better capture sophisticated feature interactions and dynamic user behaviors. The increased model complexity may slow down online inference and hinder its adoption in real-time applications. Instead, our work targets at a new model training strategy based on knowledge distillation (KD). KD is a teacher-student learning framework to transfer knowledge learned from a teacher model to a student model. The KD strategy not only allows us to simplify the student model as a vanilla DNN model but also achieves significant accuracy improvements over the state-of-the-art teacher models. The benefits thus motivate us to further explore the use of a powerful ensemble of teachers for more accurate student model training. We also propose some novel techniques to facilitate ensembled CTR prediction, including teacher gating and early stopping by distillation loss. We conduct comprehensive experiments against 12 existing models and across three industrial datasets. Both offline and online A/B testing results show the effectiveness of our KD-based training strategy.