# KEEP: An Industrial Pre-Training Framework for Online Recommendation via Knowledge Extraction and Plugging

arXiv: https://arxiv.org/abs/2208.10174 · HTML: https://mlsanity.com/paper/2208.10174/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 35 references that are in the map.
- First posted on arXiv 22 Aug 2022.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [Deep Interest Evolution Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1809.03672/) (2018)
- [S^3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization](https://mlsanity.com/paper/2008.07873/) (2020)
- [Search-based User Interest Modeling with Lifelong Sequential Behavior Data for Click-Through Rate Prediction](https://mlsanity.com/paper/2006.05639/) (2020)
- [Behavior Sequence Transformer for E-commerce Recommendation in Alibaba](https://mlsanity.com/paper/1905.06874/) (2019)
- [Practice on Long Sequential User Behavior Modeling for Click-Through Rate Prediction](https://mlsanity.com/paper/1905.09248/) (2019)
- [Learning Tree-based Deep Model for Recommender Systems](https://mlsanity.com/paper/1801.02294/) (2018)
- [Online Deep Learning: Learning Deep Neural Networks on the Fly](https://mlsanity.com/paper/1711.03705/) (2017)
- [CoNet: Collaborative Cross Networks for Cross-Domain Recommendation](https://mlsanity.com/paper/1804.06769/) (2018)
- [One Model to Serve All: Star Topology Adaptive Recommender for Multi-Domain CTR Prediction](https://mlsanity.com/paper/2101.11427/) (2021)
- [DDTCDR: Deep Dual Transfer Cross Domain Recommendation](https://mlsanity.com/paper/1910.05189/) (2019)
- [Optimized Cost per Click in Taobao Display Advertising](https://mlsanity.com/paper/1703.02091/) (2017)
- [CauseRec: Counterfactual User Sequence Synthesis for Sequential Recommendation](https://mlsanity.com/paper/2109.05261/) (2021)
- [How to Retrain Recommender System? A Sequential Meta-Learning Method](https://mlsanity.com/paper/2005.13258/) (2020)
- [DARec: Deep Domain Adaptation for Cross-Domain Recommendation via Transferring Rating Patterns](https://mlsanity.com/paper/1905.10760/) (2019)
- [TF-Ranking: Scalable TensorFlow Library for Learning-to-Rank](https://mlsanity.com/paper/1812.00073/) (2018)
- [Perceive Your Users in Depth: Learning Universal User Representations from Multiple E-commerce Tasks](https://mlsanity.com/paper/1805.10727/) (2018)
- [FeedRec: News Feed Recommendation with Various User Feedbacks](https://mlsanity.com/paper/2102.04903/) (2021)
- [PTUM: Pre-training User Model from Unlabeled User Behaviors via Self-supervision](https://mlsanity.com/paper/2010.01494/) (2020)
- [Real Negatives Matter: Continuous Training with Real Negatives for Delayed Feedback Modeling](https://mlsanity.com/paper/2104.14121/) (2021)
- [PICASSO: Unleashing the Potential of GPU-centric Training for Wide-and-deep Recommender Systems](https://mlsanity.com/paper/2204.04903/) (2022)
- [UPRec: User-Aware Pre-training for Recommender Systems](https://mlsanity.com/paper/2102.10989/) (2021)
- [User-specific Adaptive Fine-tuning for Cross-domain Recommendations](https://mlsanity.com/paper/2106.07864/) (2021)
- [CAN: Feature Co-Action for Click-Through Rate Prediction](https://mlsanity.com/paper/2011.05625/) (2020)
- [Adversarial Gradient Driven Exploration for Deep Click-Through Rate Prediction](https://mlsanity.com/paper/2112.11136/) (2021)
- [Deep Cross Networks with Aesthetic Preference for Cross-domain Recommendation](https://mlsanity.com/paper/1905.13030/) (2019)

## Built on by (most cited first, 11 of 11)

- [Enhancing Taobao Display Advertising with Multimodal Representations: Challenges, Approaches and Insights](https://mlsanity.com/paper/2407.19467/) (2024)
- [Rec4Ad: A Free Lunch to Mitigate Sample Selection Bias for Ads CTR Prediction in Taobao](https://mlsanity.com/paper/2306.03527/) (2023)
- [COPR: Consistency-Oriented Pre-Ranking for Online Advertising](https://mlsanity.com/paper/2306.03516/) (2023)
- [Capturing Conversion Rate Fluctuation during Sales Promotions: A Novel Historical Data Reuse Approach](https://mlsanity.com/paper/2305.12837/) (2023)
- [See Beyond a Single View: Multi-Attribution Learning Leads to Better Conversion Rate Prediction](https://mlsanity.com/paper/2508.15217/) (2025)
- [Improving conversion rate prediction via self-supervised pre-training in online advertising](https://mlsanity.com/paper/2401.16432/) (2024)
- [MMBee: Live Streaming Gift-Sending Recommendations via Multi-Modal Fusion and Behaviour Expansion](https://mlsanity.com/paper/2407.00056/) (2024)
- [Impression-Aware Recommender Systems](https://mlsanity.com/paper/2308.07857/) (2023)
- [Effective Two-Stage Knowledge Transfer for Multi-Entity Cross-Domain Recommendation](https://mlsanity.com/paper/2402.19101/) (2024)
- [DIIT: A Domain-Invariant Information Transfer Method for Industrial Cross-Domain Recommendation](https://mlsanity.com/paper/2410.10835/) (2024)
- [PRECTR: A Synergistic Framework for Integrating Personalized Search Relevance Matching and CTR Prediction](https://mlsanity.com/paper/2503.18395/) (2025)

## Abstract (arXiv)

An industrial recommender system generally presents a hybrid list that contains results from multiple subsystems. In practice, each subsystem is optimized with its own feedback data to avoid the disturbance among different subsystems. However, we argue that such data usage may lead to sub-optimal online performance because of the textitdata sparsity. To alleviate this issue, we propose to extract knowledge from the textitsuper-domain that contains web-scale and long-time impression data, and further assist the online recommendation task (downstream task). To this end, we propose a novel industrial KnowlEdge Extraction and Plugging (KEEP) framework, which is a two-stage framework that consists of 1) a supervised pre-training knowledge extraction module on super-domain, and 2) a plug-in network that incorporates the extracted knowledge into the downstream model. This makes it friendly for incremental training of online recommendation. Moreover, we design an efficient empirical approach for KEEP and introduce our hands-on experience during the implementation of KEEP in a large-scale industrial system. Experiments conducted on two real-world datasets demonstrate that KEEP can achieve promising results. It is notable that KEEP has also been deployed on the display advertising system in Alibaba, bringing a lift of +5.4% CTR and +4.7% RPM.