# Understanding and Scaling Collaborative Filtering Optimization from the Perspective of Matrix Rank

arXiv: https://arxiv.org/abs/2410.23300 · HTML: https://mlsanity.com/paper/2410.23300/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 48 references that are in the map.
- First posted on arXiv 15 Oct 2024.

## Builds on (most cited first, 18 of 18)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation](https://mlsanity.com/paper/2002.02126/) (2020)
- [Neural Collaborative Filtering](https://mlsanity.com/paper/1708.05031/) (2017)
- [Wide & Deep Learning for Recommender Systems](https://mlsanity.com/paper/1606.07792/) (2016)
- [Neural Graph Collaborative Filtering](https://mlsanity.com/paper/1905.08108/) (2019)
- [DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems](https://mlsanity.com/paper/2008.13535/) (2020)
- [A Survey of Graph Neural Networks for Recommender Systems: Challenges, Methods, and Directions](https://mlsanity.com/paper/2109.12843/) (2021)
- [Deep Learning based Recommender System: A Survey and New Perspectives](https://mlsanity.com/paper/1707.07435/) (2017)
- [Towards Representation Alignment and Uniformity in Collaborative Filtering](https://mlsanity.com/paper/2206.12811/) (2022)
- [Amazon-M2: A Multilingual Multi-locale Shopping Session Dataset for Recommendation and Text Generation](https://mlsanity.com/paper/2307.09688/) (2023)
- [On the Effectiveness of Sampled Softmax Loss for Item Recommendation](https://mlsanity.com/paper/2201.02327/) (2022)
- [Addressing Complex and Subjective Product-Related Queries with Customer Reviews](https://mlsanity.com/paper/1512.06863/) (2015)
- [Going Beyond Local: Global Graph-Enhanced Personalized News Recommendations](https://mlsanity.com/paper/2307.06576/) (2023)
- [Towards Mitigating Dimensional Collapse of Representations in Collaborative Filtering](https://mlsanity.com/paper/2312.17468/) (2023)
- [Toward a Better Understanding of Loss Functions for Collaborative Filtering](https://mlsanity.com/paper/2308.06091/) (2023)
- [Revisiting Recommendation Loss Functions through Contrastive Learning (Technical Report)](https://mlsanity.com/paper/2312.08520/) (2023)
- [How Does Message Passing Improve Collaborative Filtering?](https://mlsanity.com/paper/2404.08660/) (2024)

## Built on by (most cited first, 1 of 1)

- [ScienceDB AI: An LLM-Driven Agentic Recommender System for Large-Scale Scientific Data Sharing Services](https://mlsanity.com/paper/2601.01118/) (2026)

## Abstract (arXiv)

Collaborative Filtering (CF) methods dominate real-world recommender systems given their ability to learn high-quality, sparse ID-embedding tables that effectively capture user preferences. These tables scale linearly with the number of users and items, and are trained to ensure high similarity between embeddings of interacted user-item pairs, while maintaining low similarity for non-interacted pairs. Despite their high performance, encouraging dispersion for non-interacted pairs necessitates expensive regularization (e.g., negative sampling), hurting runtime and scalability. Existing research tends to address these challenges by simplifying the learning process, either by reducing model complexity or sampling data, trading performance for runtime. In this work, we move beyond model-level modifications and study the properties of the embedding tables under different learning strategies. Through theoretical analysis, we find that the singular values of the embedding tables are intrinsically linked to different CF loss functions. These findings are empirically validated on real-world datasets, demonstrating the practical benefits of higher stable rank, a continuous version of matrix rank which encodes the distribution of singular values. Based on these insights, we propose an efficient warm-start strategy that regularizes the stable rank of the user and item embeddings. We show that stable rank regularization during early training phases can promote higher-quality embeddings, resulting in training speed improvements of up to 66%. Additionally, stable rank regularization can act as a proxy for negative sampling, allowing for performance gains of up to 21% over loss functions with small negative sampling ratios. Overall, our analysis unifies current CF methods under a new perspective, their optimization of stable rank, motivating a flexible regularization method.