# Pre-train and Fine-tune: Recommenders as Large Models

arXiv: https://arxiv.org/abs/2501.14268 · HTML: https://mlsanity.com/paper/2501.14268/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 62 references that are in the map.
- First posted on arXiv 24 Jan 2025.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth](https://mlsanity.com/paper/2103.03404/) (2021)
- [Deep Interest Evolution Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1809.03672/) (2018)
- [Bias and Debias in Recommender System: A Survey and Future Directions](https://mlsanity.com/paper/2010.03240/) (2020)
- [Is ChatGPT a Good Recommender? A Preliminary Study](https://mlsanity.com/paper/2304.10149/) (2023)
- [Entire Space Multi-Task Model: An Effective Approach for Estimating Post-Click Conversion Rate](https://mlsanity.com/paper/1804.07931/) (2018)
- [A Simple Convergence Proof of Adam and Adagrad](https://mlsanity.com/paper/2003.02395/) (2020)
- [Graph Information Bottleneck](https://mlsanity.com/paper/2010.12811/) (2020)
- [Conversational Recommender System](https://mlsanity.com/paper/1806.03277/) (2018)
- [One Model to Serve All: Star Topology Adaptive Recommender for Multi-Domain CTR Prediction](https://mlsanity.com/paper/2101.11427/) (2021)
- [End-to-End User Behavior Retrieval in Click-Through RatePrediction Model](https://mlsanity.com/paper/2108.04468/) (2021)
- [Learnability for the Information Bottleneck](https://mlsanity.com/paper/1907.07331/) (2019)
- [SAR-Net: A Scenario-Aware Ranking Network for Personalized Fair Recommendation in Hundreds of Travel Scenarios](https://mlsanity.com/paper/2110.06475/) (2021)
- [Adaptive Domain Interest Network for Multi-domain Recommendation](https://mlsanity.com/paper/2206.09672/) (2022)
- [Multi-domain Recommendation with Embedding Disentangling and Domain Alignment](https://mlsanity.com/paper/2308.05508/) (2023)
- [ESMC: Entire Space Multi-Task Model for Post-Click Conversion Rate via Parameter Constraint](https://mlsanity.com/paper/2307.09193/) (2023)
- [Mitigating Mainstream Bias in Recommendation via Cost-sensitive Learning](https://mlsanity.com/paper/2307.13632/) (2023)
- [Mobile Supply: The Last Piece of Jigsaw of Recommender System](https://mlsanity.com/paper/2308.03855/) (2023)

## Built on by (most cited first, 2 of 2)

- [DeGRe: Dense-supervised Generative Reranking for Recommendation](https://mlsanity.com/paper/2605.25749/) (2026)
- [GALA: Generative Aligned Learning for Adaptive Multimodal Representation in the Taobao Shangou Recommender System](https://mlsanity.com/paper/2607.29213/) (2026)

## Abstract (arXiv)

In reality, users have different interests in different periods, regions, scenes, etc. Such changes in interest are so drastic that they are difficult to be captured by recommenders. Existing multi-domain learning can alleviate this problem. However, the structure of the industrial recommendation system is complex, the amount of data is huge, and the training cost is extremely high, so it is difficult to modify the structure of the industrial recommender and re-train it. To fill this gap, we consider recommenders as large pre-trained models and fine-tune them. We first propose the theory of the information bottleneck for fine-tuning and present an explanation for the fine-tuning technique in recommenders. To tailor for recommendation, we design an information-aware adaptive kernel (IAK) technique to fine-tune the pre-trained recommender. Specifically, we define fine-tuning as two phases: knowledge compression and knowledge matching and let the training stage of IAK explicitly approximate these two phases. Our proposed approach designed from the essence of fine-tuning is well interpretable. Extensive online and offline experiments show the superiority of our proposed method. Besides, we also share unique and important lessons we learned when deploying the method in a large-scale online platform. We also present the potential issues of fine-tuning techniques in recommendation systems and the corresponding solutions. The recommender with IAK technique has been deployed on the homepage of a billion-scale online food platform for several months and has yielded considerable profits in our business.