# Pre-trained Recommender Systems: A Causal Debiasing Perspective

arXiv: https://arxiv.org/abs/2310.19251 · HTML: https://mlsanity.com/paper/2310.19251/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 65 references that are in the map.
- First posted on arXiv 30 Oct 2023.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [KGAT: Knowledge Graph Attention Network for Recommendation](https://mlsanity.com/paper/1905.07854/) (2019)
- [Towards Universal Sequence Representation Learning for Recommender Systems](https://mlsanity.com/paper/2206.05941/) (2022)
- [Causal Intervention for Leveraging Popularity Bias in Recommendation](https://mlsanity.com/paper/2105.06067/) (2021)
- [Collaborative Deep Learning for Recommender Systems](https://mlsanity.com/paper/1409.2944/) (2014)
- [CoNet: Collaborative Cross Networks for Cross-Domain Recommendation](https://mlsanity.com/paper/1804.06769/) (2018)
- [Continuously Indexed Domain Adaptation](https://mlsanity.com/paper/2007.01807/) (2020)
- [Cross-Domain Recommendation: Challenges, Progress, and Prospects](https://mlsanity.com/paper/2103.01696/) (2021)
- [DDTCDR: Deep Dual Transfer Cross Domain Recommendation](https://mlsanity.com/paper/1910.05189/) (2019)
- [Towards Bayesian Deep Learning: A Framework and Some Existing Methods](https://mlsanity.com/paper/1608.06884/) (2016)
- [Zero-Shot Recommender Systems](https://mlsanity.com/paper/2105.08318/) (2021)
- [A Unified Approach to Domain Incremental Learning with Memory: Theory and Algorithm](https://mlsanity.com/paper/2310.12244/) (2023)
- [From Zero-Shot Learning to Cold-Start Recommendation](https://mlsanity.com/paper/1906.08511/) (2019)
- [Cross-Market Product Recommendation](https://mlsanity.com/paper/2109.05929/) (2021)
- [Domain-Indexing Variational Bayes: Interpretable Domain Index for Domain Adaptation](https://mlsanity.com/paper/2302.02561/) (2023)
- [Learning to Transfer Graph Embeddings for Inductive Graph based Recommendation](https://mlsanity.com/paper/2005.11724/) (2020)
- [A Heterogeneous Information Network based Cross Domain Insurance Recommendation System for Cold Start Users](https://mlsanity.com/paper/2007.15293/) (2020)
- [Content-aware Neural Hashing for Cold-start Recommendation](https://mlsanity.com/paper/2006.00617/) (2020)
- [Collaborative Recurrent Autoencoder: Recommend while Learning to Fill in the Blanks](https://mlsanity.com/paper/1611.00454/) (2016)
- [Joint Training Capsule Network for Cold Start Recommendation](https://mlsanity.com/paper/2005.11467/) (2020)
- [Taxonomy-Structured Domain Adaptation](https://mlsanity.com/paper/2306.07874/) (2023)
- [Personalized Federated Domain Adaptation for Item-to-Item Recommendation](https://mlsanity.com/paper/2306.03191/) (2023)

## Built on by (most cited first, 2 of 2)

- [Language-Model Prior Overcomes Cold-Start Items](https://mlsanity.com/paper/2411.09065/) (2024)
- [GRAB: An LLM-Inspired Sequence-First Click-Through Rate Prediction Modeling Paradigm](https://mlsanity.com/paper/2602.01865/) (2026)

## Abstract (arXiv)

Recent studies on pre-trained vision/language models have demonstrated the practical benefit of a new, promising solution-building paradigm in AI where models can be pre-trained on broad data describing a generic task space and then adapted successfully to solve a wide range of downstream tasks, even when training data is severely limited (e.g., in zero- or few-shot learning scenarios). Inspired by such progress, we investigate in this paper the possibilities and challenges of adapting such a paradigm to the context of recommender systems, which is less investigated from the perspective of pre-trained model. In particular, we propose to develop a generic recommender that captures universal interaction patterns by training on generic user-item interaction data extracted from different domains, which can then be fast adapted to improve few-shot learning performance in unseen new domains (with limited data). However, unlike vision/language data which share strong conformity in the semantic space, universal patterns underlying recommendation data collected across different domains (e.g., different countries or different E-commerce platforms) are often occluded by both in-domain and cross-domain biases implicitly imposed by the cultural differences in their user and item bases, as well as their uses of different e-commerce platforms. As shown in our experiments, such heterogeneous biases in the data tend to hinder the effectiveness of the pre-trained model. To address this challenge, we further introduce and formalize a causal debiasing perspective, which is substantiated via a hierarchical Bayesian deep learning model, named PreRec. Our empirical studies on real-world data show that the proposed model could significantly improve the recommendation performance in zero- and few-shot learning settings under both cross-market and cross-platform scenarios.