# The Pitfall of Scaling Up: Uncovering and Mitigating Popularity Bias Amplification in Scaling Transformer-based Recommenders

arXiv: https://arxiv.org/abs/2606.21911 · HTML: https://mlsanity.com/paper/2606.21911/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 59 of its 88 references that are in the map.
- First posted on arXiv 20 Jun 2026.

## Builds on (most cited first, 30 of 59)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [Image-based Recommendations on Styles and Substitutes](https://mlsanity.com/paper/1506.04757/) (2015)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering](https://mlsanity.com/paper/1602.01585/) (2016)
- [Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding](https://mlsanity.com/paper/1809.07426/) (2018)
- [Implicit Regularization in Deep Matrix Factorization](https://mlsanity.com/paper/1905.13655/) (2019)
- [Understanding Dimensional Collapse in Contrastive Self-supervised Learning](https://mlsanity.com/paper/2110.09348/) (2021)
- [On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization](https://mlsanity.com/paper/1802.06509/) (2018)
- [Spectral Norm Regularization for Improving the Generalizability of Deep Learning](https://mlsanity.com/paper/1705.10941/) (2017)
- [Causal Intervention for Leveraging Popularity Bias in Recommendation](https://mlsanity.com/paper/2105.06067/) (2021)
- [Learnable Item Tokenization for Generative Recommendation](https://mlsanity.com/paper/2405.07314/) (2024)
- [Model-Agnostic Counterfactual Reasoning for Eliminating Popularity Bias in Recommender System](https://mlsanity.com/paper/2010.15363/) (2020)
- [Sequential Recommendation with Graph Neural Networks](https://mlsanity.com/paper/2106.14226/) (2021)
- [Wukong: Towards a Scaling Law for Large-Scale Recommendation](https://mlsanity.com/paper/2403.02545/) (2024)
- [AutoDebias: Learning to Debias for Recommendation](https://mlsanity.com/paper/2105.04170/) (2021)
- [Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective](https://mlsanity.com/paper/2510.10150/) (2025)
- [Deconfounded Recommendation for Alleviating Bias Amplification](https://mlsanity.com/paper/2105.10648/) (2021)
- [LONGER: Scaling Up Long Sequence Modeling in Industrial Recommenders](https://mlsanity.com/paper/2505.04421/) (2025)
- [Reinforced Latent Reasoning for LLM-based Recommendation](https://mlsanity.com/paper/2505.19092/) (2025)
- [A Survey on Popularity Bias in Recommender Systems](https://mlsanity.com/paper/2308.01118/) (2023)
- [Spectral Regularization for Combating Mode Collapse in GANs](https://mlsanity.com/paper/1908.10999/) (2019)

## Abstract (arXiv)

We identify a critical pitfall in scaling transformer-based sequential recommenders: while increasing model size improves recommendation accuracy, it simultaneously amplifies popularity bias. This bias drives systems to over-recommend popular items at the expense of niche ones, which not only undermines fairness but also degrades the broader ecosystem by reinforcing the Matthew effect and filter bubbles. Consequently, this bias amplification emerges as a fundamental obstacle to sustainable model scaling. Through comprehensive theoretical and empirical analyses, we uncover the root cause of this amplification. Our findings reveal that as model depth increases, the two core components of the transformer architecture, i.e., attention aggregation and feed-forward projections, synergistically induce severe spectral collapse in model predictions, which directly translates to the amplification of popularity bias. To address this challenge, we propose SPRINT (Scalable Popularity Regularization IN Transformers), which mitigates spectral collapse during scaling by constraining (i) the maximum column-sums of the attention score matrices and (ii) the spectral norms of the feed-forward parameters. Extensive experiments demonstrate that SPRINT significantly improves both accuracy and long-tail fairness. Crucially, it yields more favorable scaling behaviors when expanding model sizes from 0.05M to 0.34B parameters. The code is available at https://github.com/Tiny-Snow/GenRec.