# PTUM: Pre-training User Model from Unlabeled User Behaviors via Self-supervision

arXiv: https://arxiv.org/abs/2010.01494 · HTML: https://mlsanity.com/paper/2010.01494/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Oct 2020.

## Built on by (most cited first, 22 of 22)

- [Recommender Systems in the Era of Large Language Models (LLMs)](https://mlsanity.com/paper/2307.02046/) (2023)
- [Learning Vector-Quantized Item Representation for Transferable Sequential Recommenders](https://mlsanity.com/paper/2210.12316/) (2022)
- [Unifying Data Perspectivism and Personalization: An Application to Social Norms](https://mlsanity.com/paper/2210.14531/) (2022)
- [A Survey of Foundation Model-Powered Recommender Systems: From Feature-Based, Generative to Agentic Paradigms](https://mlsanity.com/paper/2504.16420/) (2025)
- [KEEP: An Industrial Pre-Training Framework for Online Recommendation via Knowledge Extraction and Plugging](https://mlsanity.com/paper/2208.10174/) (2022)
- [PinFM: Foundation Model for User Activity Sequences at a Billion-scale Visual Discovery Platform](https://mlsanity.com/paper/2507.12704/) (2025)
- [Universal Item Tokenization for Transferable Generative Recommendation](https://mlsanity.com/paper/2504.04405/) (2025)
- [Empowering General-purpose User Representation with Full-life Cycle Behavior Modeling](https://mlsanity.com/paper/2110.11337/) (2021)
- [A Comprehensive Survey on Self-Supervised Learning for Recommendation](https://mlsanity.com/paper/2404.03354/) (2024)
- [Interest-oriented Universal User Representation via Contrastive Learning](https://mlsanity.com/paper/2109.08865/) (2021)
- [Learning Large-scale Universal User Representation with Sparse Mixture of Experts](https://mlsanity.com/paper/2207.04648/) (2022)
- [AdaptSSR: Pre-training User Model with Augmentation-Adaptive Self-Supervised Ranking](https://mlsanity.com/paper/2310.09706/) (2023)
- [Demographic User Modeling for Social Robotics with Multimodal Pre-trained Models](https://mlsanity.com/paper/2502.10642/) (2025)
- [PUNR: Pre-training with User Behavior Modeling for News Recommendation](https://mlsanity.com/paper/2304.12633/) (2023)
- [USE: Dynamic User Modeling with Stateful Sequence Models](https://mlsanity.com/paper/2403.13344/) (2024)
- [Abacus: Self-Supervised Event Counting-Aligned Distributional Pretraining for Sequential User Modeling](https://mlsanity.com/paper/2512.16581/) (2025)
- [UserBERT: Modeling Long- and Short-Term User Preferences via Self-Supervision](https://mlsanity.com/paper/2202.07605/) (2022)
- [PPM : A Pre-trained Plug-in Model for Click-through Rate Prediction](https://mlsanity.com/paper/2403.10049/) (2024)
- [Bootstrapping your behavior: a new pretraining strategy for user behavior sequence data](https://mlsanity.com/paper/2506.11053/) (2025)
- [Large-scale User Game Lifecycle Representation Learning](https://mlsanity.com/paper/2510.15412/) (2025)
- [Drift-Aware Multimodal User Representation Learning via Multi-Scale Temporal Modeling and Sparse Mixture-of-Experts](https://mlsanity.com/paper/2608.25773/) (2026)
- [User Representation via Cross Multi-source Behavior Pre-training for Mobile Games](https://mlsanity.com/paper/2609.01057/) (2026)

## Abstract (arXiv)

User modeling is critical for many personalized web services. Many existing methods model users based on their behaviors and the labeled data of target tasks. However, these methods cannot exploit useful information in unlabeled user behavior data, and their performance may be not optimal when labeled data is scarce. Motivated by pre-trained language models which are pre-trained on large-scale unlabeled corpus to empower many downstream tasks, in this paper we propose to pre-train user models from large-scale unlabeled user behaviors data. We propose two self-supervision tasks for user model pre-training. The first one is masked behavior prediction, which can model the relatedness between historical behaviors. The second one is next K behavior prediction, which can model the relatedness between past and future behaviors. The pre-trained user models are finetuned in downstream tasks to learn task-specific user representations. Experimental results on two real-world datasets validate the effectiveness of our proposed user model pre-training method.