# Large Language Model Empowered Recommendation Meets All-domain Continual Pre-Training

arXiv: https://arxiv.org/abs/2504.08949 · HTML: https://mlsanity.com/paper/2504.08949/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 75 references that are in the map.
- First posted on arXiv 11 Apr 2025.

## Builds on (most cited first, 30 of 43)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation](https://mlsanity.com/paper/2305.00447/) (2023)
- [Large Language Models are Zero-Shot Rankers for Recommender Systems](https://mlsanity.com/paper/2305.08845/) (2023)
- [Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender System](https://mlsanity.com/paper/2303.14524/) (2023)
- [Continual Learning of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2404.16789/) (2024)
- [Recommendation as Instruction Following: A Large Language Model Empowered Recommendation Approach](https://mlsanity.com/paper/2305.07001/) (2023)
- [Text Is All You Need: Learning Language Representations for Sequential Recommendation](https://mlsanity.com/paper/2305.13731/) (2023)
- [Continual Pre-Training of Large Language Models: How to (re)warm your model?](https://mlsanity.com/paper/2308.04014/) (2023)
- [Where to Go Next for Recommender Systems? ID- vs. Modality-based Recommender Models Revisited](https://mlsanity.com/paper/2303.13835/) (2023)
- [Towards Universal Sequence Representation Learning for Recommender Systems](https://mlsanity.com/paper/2206.05941/) (2022)
- [CoLLM: Integrating Collaborative Embeddings into Large Language Models for Recommendation](https://mlsanity.com/paper/2310.19488/) (2023)
- [Towards Open-World Recommendation with Knowledge Augmentation from Large Language Models](https://mlsanity.com/paper/2306.10933/) (2023)
- [Wukong: Towards a Scaling Law for Large-Scale Recommendation](https://mlsanity.com/paper/2403.02545/) (2024)
- [On Softmax Direct Preference Optimization for Recommendation](https://mlsanity.com/paper/2406.09215/) (2024)
- [LLaRA: Large Language-Recommendation Assistant](https://mlsanity.com/paper/2312.02445/) (2023)
- [Understanding Scaling Laws for Recommendation Models](https://mlsanity.com/paper/2208.08489/) (2022)
- [Plug-in Diffusion Model for Sequential Recommendation](https://mlsanity.com/paper/2401.02913/) (2024)
- [Efficient Continual Pre-training by Mitigating the Stability Gap](https://mlsanity.com/paper/2406.14833/) (2024)
- [Scaling Law of Large Sequential Recommendation Models](https://mlsanity.com/paper/2311.11351/) (2023)
- [Breaking the Length Barrier: LLM-Enhanced CTR Prediction in Long Textual User Behaviors](https://mlsanity.com/paper/2403.19347/) (2024)
- [NineRec: A Benchmark Dataset Suite for Evaluating Transferable Recommendation](https://mlsanity.com/paper/2309.07705/) (2023)

## Built on by (most cited first, 4 of 4)

- [PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning](https://mlsanity.com/paper/2601.13020/) (2026)
- [Continual Learning in Transition](https://mlsanity.com/paper/2608.06216/) (2026)
- [EchoRec: Multi-Item Prediction-Empowered Generative Recommendation via Cycle-Consistent Preference Alignment](https://mlsanity.com/paper/2608.14011/) (2026)
- [Preference Flow Matching with Spectral Factorization for Micro-video Recommendation](https://mlsanity.com/paper/2608.26579/) (2026)

## Abstract (arXiv)

Recent research efforts have investigated how to integrate Large Language Models (LLMs) into recommendation, capitalizing on their semantic comprehension and open-world knowledge for user behavior understanding. These approaches predominantly employ supervised fine-tuning on single-domain user interactions to adapt LLMs for specific recommendation tasks. However, they typically encounter dual challenges: the mismatch between general language representations and domain-specific preference patterns, as well as the limited adaptability to multi-domain recommendation scenarios. To bridge these gaps, we introduce CPRec -- an All-domain Continual Pre-Training framework for Recommendation -- designed to holistically align LLMs with universal user behaviors through the continual pre-training paradigm. Specifically, we first design a unified prompt template and organize users' multi-domain behaviors into domain-specific behavioral sequences and all-domain mixed behavioral sequences that emulate real-world user decision logic. To optimize behavioral knowledge infusion, we devise a Warmup-Stable-Annealing learning rate schedule tailored for the continual pre-training paradigm in recommendation to progressively enhance the LLM's capability in knowledge adaptation from open-world knowledge to universal recommendation tasks. To evaluate the effectiveness of our CPRec, we implement it on a large-scale dataset covering seven domains and conduct extensive experiments on five real-world datasets from two distinct platforms. Experimental results confirm that our continual pre-training paradigm significantly mitigates the semantic-behavioral discrepancy and achieves state-of-the-art performance in all recommendation scenarios. The source code will be released upon acceptance.