# On the Role of Weight Decay in Collaborative Filtering: A Popularity Perspective

arXiv: https://arxiv.org/abs/2505.11318 · HTML: https://mlsanity.com/paper/2505.11318/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 44 references that are in the map.
- First posted on arXiv 16 May 2025.

## Builds on (most cited first, 21 of 21)

- [LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation](https://mlsanity.com/paper/2002.02126/) (2020)
- [Neural Collaborative Filtering](https://mlsanity.com/paper/1708.05031/) (2017)
- [Wide & Deep Learning for Recommender Systems](https://mlsanity.com/paper/1606.07792/) (2016)
- [Neural Graph Collaborative Filtering](https://mlsanity.com/paper/1905.08108/) (2019)
- [DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems](https://mlsanity.com/paper/2008.13535/) (2020)
- [A Survey of Graph Neural Networks for Recommender Systems: Challenges, Methods, and Directions](https://mlsanity.com/paper/2109.12843/) (2021)
- [Deep Learning based Recommender System: A Survey and New Perspectives](https://mlsanity.com/paper/1707.07435/) (2017)
- [Neural Collaborative Filtering vs. Matrix Factorization Revisited](https://mlsanity.com/paper/2005.09683/) (2020)
- [Model-Agnostic Counterfactual Reasoning for Eliminating Popularity Bias in Recommender System](https://mlsanity.com/paper/2010.15363/) (2020)
- [AutoDebias: Learning to Debias for Recommendation](https://mlsanity.com/paper/2105.04170/) (2021)
- [A Survey on Popularity Bias in Recommender Systems](https://mlsanity.com/paper/2308.01118/) (2023)
- [On the Effectiveness of Sampled Softmax Loss for Item Recommendation](https://mlsanity.com/paper/2201.02327/) (2022)
- [Addressing Complex and Subjective Product-Related Queries with Customer Reviews](https://mlsanity.com/paper/1512.06863/) (2015)
- [How Do Recommendation Models Amplify Popularity Bias? An Analysis from the Spectral Perspective](https://mlsanity.com/paper/2404.12008/) (2024)
- [Adap-τ: Adaptively Modulating Embedding Magnitude for Recommendation](https://mlsanity.com/paper/2302.04775/) (2023)
- [Investigating Accuracy-Novelty Performance for Graph-based Collaborative Filtering](https://mlsanity.com/paper/2204.12326/) (2022)
- [Going Beyond Local: Global Graph-Enhanced Personalized News Recommendations](https://mlsanity.com/paper/2307.06576/) (2023)
- [Towards Mitigating Dimensional Collapse of Representations in Collaborative Filtering](https://mlsanity.com/paper/2312.17468/) (2023)
- [Toward a Better Understanding of Loss Functions for Collaborative Filtering](https://mlsanity.com/paper/2308.06091/) (2023)
- [The Hidden Pitfalls of the Cosine Similarity Loss](https://mlsanity.com/paper/2406.16468/) (2024)
- [Popularity Debiasing from Exposure to Interaction in Collaborative Filtering](https://mlsanity.com/paper/2305.05204/) (2023)

## Built on by (most cited first, 2 of 2)

- [AgentDR: Dynamic Recommendation with Implicit Item-Item Relations via LLM-based Agents](https://mlsanity.com/paper/2510.05598/) (2025)
- [Rethinking Popularity Bias in Collaborative Filtering via Analytical Vector Decomposition](https://mlsanity.com/paper/2512.10688/) (2025)

## Abstract (arXiv)

Collaborative filtering (CF) enables large-scale recommendation systems by encoding information from historical user-item interactions into dense ID-embedding tables. However, as embedding tables grow, closed-form solutions become impractical, often necessitating the use of mini-batch gradient descent for training. Despite extensive work on designing loss functions to train CF models, we argue that one core component of these pipelines is heavily overlooked: weight decay. Attaining high-performing models typically requires careful tuning of weight decay, regardless of loss, yet its necessity is not well understood. In this work, we question why weight decay is crucial in CF pipelines and how it impacts training. Through theoretical and empirical analysis, we surprisingly uncover that weight decay's primary function is to encode popularity information into the magnitudes of the embedding vectors. Moreover, we find that tuning weight decay acts as a coarse, non-linear knob to influence preference towards popular or unpopular items. Based on these findings, we propose PRISM (Popularity-awaRe Initialization Strategy for embedding Magnitudes), a straightforward yet effective solution to simplify the training of high-performing CF models. PRISM pre-encodes the popularity information typically learned through weight decay, eliminating its necessity. Our experiments show that PRISM improves performance by up to 4.77% and reduces training times by 38.48%, compared to state-of-the-art training strategies. Additionally, we parameterize PRISM to modulate the initialization strength, offering a cost-effective and meaningful strategy to mitigate popularity bias.