# BehaveGPT: A Foundation Model for Large-scale User Behavior Modeling

arXiv: https://arxiv.org/abs/2505.17631 · HTML: https://mlsanity.com/paper/2505.17631/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 59 references that are in the map.
- First posted on arXiv 23 May 2025.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [One Fits All:Power General Time Series Analysis by Pretrained LM](https://mlsanity.com/paper/2302.11939/) (2023)
- [DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models](https://mlsanity.com/paper/2210.08933/) (2022)
- [SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient](https://mlsanity.com/paper/1609.05473/) (2016)
- [Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectives](https://mlsanity.com/paper/2312.11970/) (2023)
- [Statistical Aspects of Wasserstein Distances](https://mlsanity.com/paper/1806.05500/) (2018)
- [Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations](https://mlsanity.com/paper/2402.17152/) (2024)
- [UniST: A Prompt-Empowered Universal Model for Urban Spatio-Temporal Prediction](https://mlsanity.com/paper/2402.11838/) (2024)
- [Wukong: Towards a Scaling Law for Large-Scale Recommendation](https://mlsanity.com/paper/2403.02545/) (2024)
- [Simplify and Robustify Negative Sampling for Implicit Collaborative Filtering](https://mlsanity.com/paper/2009.03376/) (2020)
- [Long-tail Session-based Recommendation](https://mlsanity.com/paper/2007.12329/) (2020)
- [A Survey of Large Language Model Empowered Agents for Recommendation and Search: Towards Next-Generation Information Retrieval](https://mlsanity.com/paper/2503.05659/) (2025)
- [Session-based Recommendation with Hypergraph Attention Networks](https://mlsanity.com/paper/2112.14266/) (2021)
- [IART: Intent-aware Response Ranking with Transformers in Information-seeking Conversation Systems](https://mlsanity.com/paper/2002.00571/) (2020)
- [Privacy-Preserving Synthetic Data Generation for Recommendation Systems](https://mlsanity.com/paper/2209.13133/) (2022)
- [Empowering Long-tail Item Recommendation through Cross Decoupling Network (CDN)](https://mlsanity.com/paper/2210.14309/) (2022)
- [Diffusion-Based Cloud-Edge-Device Collaborative Learning for Next POI Recommendations](https://mlsanity.com/paper/2405.13811/) (2024)
- [Cold-start Sequential Recommendation via Meta Learner](https://mlsanity.com/paper/2012.05462/) (2020)
- [Alleviating Video-Length Effect for Micro-video Recommendation](https://mlsanity.com/paper/2308.14276/) (2023)
- [A Population-to-individual Tuning Framework for Adapting Pretrained LM to On-device User Intent Prediction](https://mlsanity.com/paper/2408.09815/) (2024)
- [Intent-aware Ranking Ensemble for Personalized Recommendation](https://mlsanity.com/paper/2304.07450/) (2023)

## Built on by (most cited first, 2 of 2)

- [LLMs Reading the Rhythms of Daily Life: Aligned Understanding for Behavior Prediction and Generation](https://mlsanity.com/paper/2604.23578/) (2026)
- [Socio-Foundation: A Model for Generalizable Individual Behavior Simulation via Hierarchical Capability Distillation](https://mlsanity.com/paper/2610.08967/) (2026)

## Abstract (arXiv)

In recent years, foundational models have revolutionized the fields of language and vision, demonstrating remarkable abilities in understanding and generating complex data; however, similar advances in user behavior modeling have been limited, largely due to the complexity of behavioral data and the challenges involved in capturing intricate temporal and contextual relationships in user activities. To address this, we propose BehaveGPT, a foundational model designed specifically for large-scale user behavior prediction. Leveraging transformer-based architecture and a novel pretraining paradigm, BehaveGPT is trained on vast user behavior datasets, allowing it to learn complex behavior patterns and support a range of downstream tasks, including next behavior prediction, long-term generation, and cross-domain adaptation. Our approach introduces the DRO-based pretraining paradigm tailored for user behavior data, which improves model generalization and transferability by equitably modeling both head and tail behaviors. Extensive experiments on real-world datasets demonstrate that BehaveGPT outperforms state-of-the-art baselines, achieving more than a 10% improvement in macro and weighted recall, showcasing its ability to effectively capture and predict user behavior. Furthermore, we measure the scaling law in the user behavior domain for the first time on the Honor dataset, providing insights into how model performance scales with increased data and parameter sizes.