# PPM : A Pre-trained Plug-in Model for Click-through Rate Prediction

arXiv: https://arxiv.org/abs/2403.10049 · HTML: https://mlsanity.com/paper/2403.10049/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 15 Mar 2024.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [Wide & Deep Learning for Recommender Systems](https://mlsanity.com/paper/1606.07792/) (2016)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [Deep Interest Evolution Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1809.03672/) (2018)
- [VBPR: Visual Bayesian Personalized Ranking from Implicit Feedback](https://mlsanity.com/paper/1510.01784/) (2015)
- [Towards Universal Sequence Representation Learning for Recommender Systems](https://mlsanity.com/paper/2206.05941/) (2022)
- [Cross-Domain Recommendation: Challenges, Progress, and Prospects](https://mlsanity.com/paper/2103.01696/) (2021)
- [Item2Vec: Neural Item Embedding for Collaborative Filtering](https://mlsanity.com/paper/1603.04259/) (2016)
- [Open Benchmarking for Click-Through Rate Prediction](https://mlsanity.com/paper/2009.05794/) (2020)
- [Non-invasive Self-attention for Side Information Fusion in Sequential Recommendation](https://mlsanity.com/paper/2103.03578/) (2021)
- [ItemSage: Learning Product Embeddings for Shopping Recommendations at Pinterest](https://mlsanity.com/paper/2205.11728/) (2022)
- [PTUM: Pre-training User Model from Unlabeled User Behaviors via Self-supervision](https://mlsanity.com/paper/2010.01494/) (2020)
- [UPRec: User-Aware Pre-training for Recommender Systems](https://mlsanity.com/paper/2102.10989/) (2021)
- [Multi-Interactive Attention Network for Fine-grained Feature Learning in CTR Prediction](https://mlsanity.com/paper/2012.06968/) (2020)
- [MvFS: Multi-view Feature Selection for Recommender System](https://mlsanity.com/paper/2309.02064/) (2023)
- [Deep Interest Highlight Network for Click-Through Rate Prediction in Trigger-Induced Recommendation](https://mlsanity.com/paper/2202.08959/) (2022)
- [MAKE: Vision-Language Pre-training based Product Retrieval in Taobao Search](https://mlsanity.com/paper/2301.12646/) (2023)
- [UserBERT: Contrastive User Model Pre-training](https://mlsanity.com/paper/2109.01274/) (2021)
- [HIEN: Hierarchical Intention Embedding Network for Click-Through Rate Prediction](https://mlsanity.com/paper/2206.00510/) (2022)

## Abstract (arXiv)

Click-through rate (CTR) prediction is a core task in recommender systems. Existing methods (IDRec for short) rely on unique identities to represent distinct users and items that have prevailed for decades. On one hand, IDRec often faces significant performance degradation on cold-start problem; on the other hand, IDRec cannot use longer training data due to constraints imposed by iteration efficiency. Most prior studies alleviate the above problems by introducing pre-trained knowledge(e.g. pre-trained user model or multi-modal embeddings). However, the explosive growth of online latency can be attributed to the huge parameters in the pre-trained model. Therefore, most of them cannot employ the unified model of end-to-end training with IDRec in industrial recommender systems, thus limiting the potential of the pre-trained model. To this end, we propose a Pre-trained Plug-in CTR Model, namely PPM. PPM employs multi-modal features as input and utilizes large-scale data for pre-training. Then, PPM is plugged in IDRec model to enhance unified model's performance and iteration efficiency. Upon incorporating IDRec model, certain intermediate results within the network are cached, with only a subset of the parameters participating in training and serving. Hence, our approach can successfully deploy an end-to-end model without causing huge latency increases. Comprehensive offline experiments and online A/B testing at JD E-commerce demonstrate the efficiency and effectiveness of PPM.