# Evaluating Performance and Bias of Negative Sampling in Large-Scale Sequential Recommendation Models

arXiv: https://arxiv.org/abs/2410.17276 · HTML: https://mlsanity.com/paper/2410.17276/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 38 references that are in the map.
- First posted on arXiv 8 Oct 2024.

## Builds on (most cited first, 24 of 24)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding](https://mlsanity.com/paper/1809.07426/) (2018)
- [Recurrent Neural Networks with Top-k Gains for Session-based Recommendations](https://mlsanity.com/paper/1706.03847/) (2017)
- [Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation](https://mlsanity.com/paper/2110.05730/) (2021)
- [Filter-enhanced MLP is All You Need for Sequential Recommendation](https://mlsanity.com/paper/2202.13556/) (2022)
- [Managing Popularity Bias in Recommender Systems with Personalized Re-ranking](https://mlsanity.com/paper/1901.07555/) (2019)
- [The Unfairness of Popularity Bias in Recommendation](https://mlsanity.com/paper/1907.13286/) (2019)
- [Frequency Enhanced Hybrid Attention Network for Sequential Recommendation](https://mlsanity.com/paper/2304.09184/) (2023)
- [Exploring Data Splitting Strategies for the Evaluation of Recommendation Models](https://mlsanity.com/paper/2007.13237/) (2020)
- [gSASRec: Reducing Overconfidence in Sequential Recommendation Trained with Negative Sampling](https://mlsanity.com/paper/2308.07192/) (2023)
- [TransRec: Learning Transferable Recommendation from Mixture-of-Modality Feedback](https://mlsanity.com/paper/2206.06190/) (2022)
- [A Case Study on Sampling Strategies for Evaluating Neural Sequential Item Recommendation Models](https://mlsanity.com/paper/2107.13045/) (2021)
- [Cross-Batch Negative Sampling for Training Two-Tower Recommenders](https://mlsanity.com/paper/2110.15154/) (2021)
- [A Systematic Review and Replicability Study of BERT4Rec for Sequential Recommendation](https://mlsanity.com/paper/2207.07483/) (2022)
- [Fairly Adaptive Negative Sampling for Recommendations](https://mlsanity.com/paper/2302.08266/) (2023)
- [Widespread Flaws in Offline Evaluation of Recommender Systems](https://mlsanity.com/paper/2307.14951/) (2023)
- [Cache-Augmented Inbatch Importance Resampling for Training Recommender Retriever](https://mlsanity.com/paper/2205.14859/) (2022)
- [Generating Negative Samples for Sequential Recommendation](https://mlsanity.com/paper/2208.03645/) (2022)
- [Towards Automated Negative Sampling in Implicit Recommendation](https://mlsanity.com/paper/2311.03526/) (2023)

## Built on by (most cited first, 4 of 4)

- [Contrastive Learning for Diversity-Aware Product Recommendations in Retail](https://mlsanity.com/paper/2602.08886/) (2026)
- [Aligning Recommendations with User Popularity Preferences](https://mlsanity.com/paper/2604.01036/) (2026)
- [Learning over Positive and Negative Edges with Contrastive Message Passing](https://mlsanity.com/paper/2605.17854/) (2026)
- [Training seeds and model-selection stability in recommender-system evaluation](https://mlsanity.com/paper/2609.02499/) (2026)

## Abstract (arXiv)

Large-scale industrial recommendation models predict the most relevant items from catalogs containing millions or billions of options. To train these models efficiently, a small set of irrelevant items (negative samples) is selected from the vast catalog for each relevant item (positive example), helping the model distinguish between relevant and irrelevant items. Choosing the right negative sampling method is a common challenge. We address this by implementing and comparing various negative sampling methods - random, popularity-based, in-batch, mixed, adaptive, and adaptive with mixed variants - on modern sequential recommendation models. Our experiments, including hyperparameter optimization and 20x repeats on three benchmark datasets with varying popularity biases, show how the choice of method and dataset characteristics impact key model performance metrics. We also reveal that average performance metrics often hide imbalances across popularity bands (head, mid, tail). We find that commonly used random negative sampling reinforces popularity bias and performs best for head items. Popularity-based methods (in-batch and global popularity negative sampling) can offer balanced performance at the cost of lower overall model performance results. Our study serves as a practical guide to the trade-offs in selecting a negative sampling method for large-scale sequential recommendation models. Code, datasets, experimental results and hyperparameters are available at: https://github.com/apple/ml-negative-sampling.