# Scaling Sequential Recommendation Models with Transformers

arXiv: https://arxiv.org/abs/2412.07585 · HTML: https://mlsanity.com/paper/2412.07585/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 61 references that are in the map.
- First posted on arXiv 10 Dec 2024.

## Builds on (most cited first, 30 of 35)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting](https://mlsanity.com/paper/1912.09363/) (2019)
- [Image-based Recommendations on Styles and Substitutes](https://mlsanity.com/paper/1506.04757/) (2015)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering](https://mlsanity.com/paper/1602.01585/) (2016)
- [Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding](https://mlsanity.com/paper/1809.07426/) (2018)
- [S^3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization](https://mlsanity.com/paper/2008.07873/) (2020)
- [Unified Scaling Laws for Routed Language Models](https://mlsanity.com/paper/2202.01169/) (2022)
- [Neural Attentive Session-based Recommendation](https://mlsanity.com/paper/1711.04725/) (2017)
- [A Constructive Prediction of the Generalization Error Across Scales](https://mlsanity.com/paper/1909.12673/) (2019)
- [Recommender Systems in the Era of Large Language Models (LLMs)](https://mlsanity.com/paper/2307.02046/) (2023)
- [Fusing Similarity Models with Markov Chains for Sparse Sequential Recommendation](https://mlsanity.com/paper/1609.09152/) (2016)
- [RecBole: Towards a Unified, Comprehensive and Efficient Framework for Recommendation Algorithms](https://mlsanity.com/paper/2011.01731/) (2020)
- [Where to Go Next for Recommender Systems? ID- vs. Modality-based Recommender Models Revisited](https://mlsanity.com/paper/2303.13835/) (2023)
- [Intent Contrastive Learning for Sequential Recommendation](https://mlsanity.com/paper/2202.02519/) (2022)
- [Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design](https://mlsanity.com/paper/2305.13035/) (2023)
- [Improved Recurrent Neural Networks for Session-based Recommendations](https://mlsanity.com/paper/1606.08117/) (2016)
- [Translation-based Recommendation](https://mlsanity.com/paper/1707.02410/) (2017)
- [Contrastive Self-supervised Sequential Recommendation with Robust Augmentation](https://mlsanity.com/paper/2108.06479/) (2021)
- [On the Effectiveness of Sampled Softmax Loss for Item Recommendation](https://mlsanity.com/paper/2201.02327/) (2022)
- [Transformers in Speech Processing: A Survey](https://mlsanity.com/paper/2303.11607/) (2023)
- [Understanding Scaling Laws for Recommendation Models](https://mlsanity.com/paper/2208.08489/) (2022)
- [Scaling Law for Recommendation Models: Towards General-purpose User Representations](https://mlsanity.com/paper/2111.11294/) (2021)
- [Lightweight Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/2108.11333/) (2021)
- [A Systematic Review and Replicability Study of BERT4Rec for Sequential Recommendation](https://mlsanity.com/paper/2207.07483/) (2022)

## Built on by (most cited first, 7 of 7)

- [Adaptive Weighted Loss for Sequential Recommendations on Sparse Domains](https://mlsanity.com/paper/2510.04375/) (2025)
- [STARS: Semantic Tokens with Augmented Representations for Recommendation at Scale](https://mlsanity.com/paper/2512.10149/) (2025)
- [ULTRA:Urdu Language Transformer-based Recommendation Architecture](https://mlsanity.com/paper/2602.11836/) (2026)
- [The Pitfall of Scaling Up: Uncovering and Mitigating Popularity Bias Amplification in Scaling Transformer-based Recommenders](https://mlsanity.com/paper/2606.21911/) (2026)
- [Towards a Densing Law for User Representation Learning at Billion-Scale Capacity](https://mlsanity.com/paper/2608.23392/) (2026)
- [CRAMER: Control via Request-Aware Masking for Editing Recommenders](https://mlsanity.com/paper/2608.25370/) (2026)
- [DP-Rec: Towards Dynamic Patching for Efficient Long-Sequence Recommendation](https://mlsanity.com/paper/2609.32215/) (2026)

## Abstract (arXiv)

Modeling user preferences has been mainly addressed by looking at users' interaction history with the different elements available in the system. Tailoring content to individual preferences based on historical data is the main goal of sequential recommendation. The nature of the problem, as well as the good performance observed across various domains, has motivated the use of the transformer architecture, which has proven effective in leveraging increasingly larger amounts of training data when accompanied by an increase in the number of model parameters. This scaling behavior has brought a great deal of attention, as it provides valuable guidance in the design and training of even larger models. Taking inspiration from the scaling laws observed in training large language models, we explore similar principles for sequential recommendation. We use the full Amazon Product Data dataset, which has only been partially explored in other studies, and reveal scaling behaviors similar to those found in language models. Compute-optimal training is possible but requires a careful analysis of the compute-performance trade-offs specific to the application. We also show that performance scaling translates to downstream tasks by fine-tuning larger pre-trained models on smaller task-specific domains. Our approach and findings provide a strategic roadmap for model training and deployment in real high-dimensional preference spaces, facilitating better training and inference efficiency. We hope this paper bridges the gap between the potential of transformers and the intrinsic complexities of high-dimensional sequential recommendation in real-world recommender systems. Code and models can be found at https://github.com/mercadolibre/srt