# Scaling Transformers for Discriminative Recommendation via Generative Pretraining

arXiv: https://arxiv.org/abs/2506.03699 · HTML: https://mlsanity.com/paper/2506.03699/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jun 2025.

## Built on by (most cited first, 10 of 10)

- [MMQ: Multimodal Mixture-of-Quantization Tokenization for Semantic ID Generation and User Behavioral Adaptation](https://mlsanity.com/paper/2508.15281/) (2025)
- [GR-LLMs: Recent Advances in Generative Recommendation Based on Large Language Models](https://mlsanity.com/paper/2507.06507/) (2025)
- [From Scaling to Structured Expressivity: Rethinking Transformers for CTR Prediction](https://mlsanity.com/paper/2511.12081/) (2025)
- [Equip Pre-ranking with Target Attention by Residual Quantization](https://mlsanity.com/paper/2509.16931/) (2025)
- [Adaptive Regularization for Large-Scale Sparse Feature Embedding Models](https://mlsanity.com/paper/2511.06374/) (2025)
- [xGR: Efficient Generative Recommendation Serving at Scale](https://mlsanity.com/paper/2512.11529/) (2025)
- [Closing the Auto-Research Loop: An AI Co-Scientist for Production Search Ranking](https://mlsanity.com/paper/2603.22376/) (2026)
- [Expand More, Shrink Less: Shaping Effective-Rank Dynamics for Dense Scaling in Recommendation](https://mlsanity.com/paper/2605.23191/) (2026)
- [Long-term User Engagement Optimization through Model-agnostic Downstream Rewards Learning](https://mlsanity.com/paper/2607.14192/) (2026)
- [SIGMA: A Semantic-Grounded Instruction-Driven Generative Multi-Task Recommender at AliExpress](https://mlsanity.com/paper/2602.22913/) (2026)

## Abstract (arXiv)

Discriminative recommendation tasks, such as CTR (click-through rate) and CVR (conversion rate) prediction, play critical roles in the ranking stage of large-scale industrial recommender systems. However, training a discriminative model encounters a significant overfitting issue induced by data sparsity. Moreover, this overfitting issue worsens with larger models, causing them to underperform smaller ones. To address the overfitting issue and enhance model scalability, we propose a framework named GPSD (Generative Pretraining for Scalable Discriminative Recommendation), drawing inspiration from generative training, which exhibits no evident signs of overfitting. GPSD leverages the parameters learned from a pretrained generative model to initialize a discriminative model, and subsequently applies a sparse parameter freezing strategy. Extensive experiments conducted on both industrial-scale and publicly available datasets demonstrate the superior performance of GPSD. Moreover, it delivers remarkable improvements in online A/B tests. GPSD offers two primary advantages: 1) it substantially narrows the generalization gap in model training, resulting in better test performance; and 2) it leverages the scalability of Transformers, delivering consistent performance gains as models are scaled up. Specifically, we observe consistent performance improvements as the model dense parameters scale from 13K to 0.3B, closely adhering to power laws. These findings pave the way for unifying the architectures of recommendation models and language models, enabling the direct application of techniques well-established in large language models to recommendation models. The code is available at https://github.com/chqiwang/gpsd-rec.