# User Representation via Cross Multi-source Behavior Pre-training for Mobile Games

arXiv: https://arxiv.org/abs/2609.01057 · HTML: https://mlsanity.com/paper/2609.01057/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 57 references that are in the map.
- First posted on arXiv 1 Sep 2026.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://mlsanity.com/paper/1703.04247/) (2017)
- [R-Drop: Regularized Dropout for Neural Networks](https://mlsanity.com/paper/2106.14448/) (2021)
- [Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation](https://mlsanity.com/paper/2110.05730/) (2021)
- [Contrastive Self-supervised Sequential Recommendation with Robust Augmentation](https://mlsanity.com/paper/2108.06479/) (2021)
- [PLIP: Language-Image Pre-training for Person Representation Learning](https://mlsanity.com/paper/2305.08386/) (2023)
- [ATRank: An Attention-Based User Behavior Modeling Framework for Recommendation](https://mlsanity.com/paper/1711.06632/) (2017)
- [DA-GCN: A Domain-aware Attentive Graph Convolution Network for Shared-account Cross-domain Sequential Recommendation](https://mlsanity.com/paper/2105.03300/) (2021)
- [Perceive Your Users in Depth: Learning Universal User Representations from Multiple E-commerce Tasks](https://mlsanity.com/paper/1805.10727/) (2018)
- [PTUM: Pre-training User Model from Unlabeled User Behaviors via Self-supervision](https://mlsanity.com/paper/2010.01494/) (2020)
- [Beyond Clicks: Modeling Multi-Relational Item Graph for Session-Based Target Behavior Prediction](https://mlsanity.com/paper/2002.07993/) (2020)
- [Multi-view Multi-behavior Contrastive Learning in Recommendation](https://mlsanity.com/paper/2203.10576/) (2022)
- [Adaptive Domain Interest Network for Multi-domain Recommendation](https://mlsanity.com/paper/2206.09672/) (2022)
- [Triple Sequence Learning for Cross-domain Recommendation](https://mlsanity.com/paper/2304.05027/) (2023)
- [UPRec: User-Aware Pre-training for Recommender Systems](https://mlsanity.com/paper/2102.10989/) (2021)
- [Kalman Filtering Attention for User Behavior Modeling in CTR Prediction](https://mlsanity.com/paper/2010.00985/) (2020)
- [PerFedRec++: Enhancing Personalized Federated Recommendation with Self-Supervised Pre-Training](https://mlsanity.com/paper/2305.06622/) (2023)
- [Multi-behavior Self-supervised Learning for Recommendation](https://mlsanity.com/paper/2305.18238/) (2023)
- [User-specific Adaptive Fine-tuning for Cross-domain Recommendations](https://mlsanity.com/paper/2106.07864/) (2021)
- [Federated Graph Learning for Cross-Domain Recommendation](https://mlsanity.com/paper/2410.08249/) (2024)

## Abstract (arXiv)

User representation pre-training has become a fundamental paradigm for alleviating data sparsity in downstream personalization tasks. However, existing studies predominantly focus on single-app or app-level behaviors, overlooking the inherently cross-source and multi-granular nature of user activities on mobile devices. At the device level, user intent emerges from complex interactions among heterogeneous behavior sources and hierarchical action structures, posing challenges that cannot be addressed by conventional app-centric modeling. To tackle this issue, we propose CM-PTM, a novel Cross Multi-source Behavior Pre-Training Model tailored for mobile game user representation learning on device-level behavioral logs. CM-PTM employs hierarchical cascaded mask-then-predict proxy tasks that first infer the source of the next behavior and then progressively refine predictions at the app-action level. This design enables unified modeling of cross-source dependencies and fine-grained behavioral dynamics within a single pre-training paradigm. Extensive experiments on large-scale real-world mobile datasets demonstrate that CM-PTM effectively captures users' endogenous interests and consistently delivers significant performance gains on downstream mobile game recommendation tasks.