# Scalable Cross-Entropy Loss for Sequential Recommendations with Large Item Catalogs

arXiv: https://arxiv.org/abs/2409.18721 · HTML: https://mlsanity.com/paper/2409.18721/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 68 references that are in the map.
- First posted on arXiv 27 Sep 2024.

## Builds on (most cited first, 30 of 31)

- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [Image-based Recommendations on Styles and Substitutes](https://mlsanity.com/paper/1506.04757/) (2015)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding](https://mlsanity.com/paper/1809.07426/) (2018)
- [Recurrent Neural Networks with Top-k Gains for Session-based Recommendations](https://mlsanity.com/paper/1706.03847/) (2017)
- [Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation](https://mlsanity.com/paper/2110.05730/) (2021)
- [Practical and Optimal LSH for Angular Distance](https://mlsanity.com/paper/1509.02897/) (2015)
- [Yelp Dataset Challenge: Review Rating Prediction](https://mlsanity.com/paper/1605.05362/) (2016)
- [Contrastive Self-supervised Sequential Recommendation with Robust Augmentation](https://mlsanity.com/paper/2108.06479/) (2021)
- [Frequency Enhanced Hybrid Attention Network for Sequential Recommendation](https://mlsanity.com/paper/2304.09184/) (2023)
- [Exploring Data Splitting Strategies for the Evaluation of Recommendation Models](https://mlsanity.com/paper/2007.13237/) (2020)
- [A Critical Study on Data Leakage in Recommender System Offline Evaluation](https://mlsanity.com/paper/2010.11060/) (2020)
- [Non-invasive Self-attention for Side Information Fusion in Sequential Recommendation](https://mlsanity.com/paper/2103.03578/) (2021)
- [Simplify and Robustify Negative Sampling for Implicit Collaborative Filtering](https://mlsanity.com/paper/2009.03376/) (2020)
- [Turning Dross Into Gold Loss: is BERT4Rec really better than SASRec?](https://mlsanity.com/paper/2309.07602/) (2023)
- [Quantization based Fast Inner Product Search](https://mlsanity.com/paper/1509.01469/) (2015)
- [gSASRec: Reducing Overconfidence in Sequential Recommendation Trained with Negative Sampling](https://mlsanity.com/paper/2308.07192/) (2023)
- [A Case Study on Sampling Strategies for Evaluating Neural Sequential Item Recommendation Models](https://mlsanity.com/paper/2107.13045/) (2021)
- [Cross-Batch Negative Sampling for Training Two-Tower Recommenders](https://mlsanity.com/paper/2110.15154/) (2021)
- [Vista: A Visually, Socially, and Temporally-aware Model for Artistic Recommendation](https://mlsanity.com/paper/1607.04373/) (2016)
- [Denoising Self-attentive Sequential Recommendation](https://mlsanity.com/paper/2212.04120/) (2022)
- [A New Unbiased and Efficient Class of LSH-Based Samplers and Estimators for Partition Function Computation in Log-Linear Models](https://mlsanity.com/paper/1703.05160/) (2017)
- [Contrastive Learning with Bidirectional Transformers for Sequential Recommendation](https://mlsanity.com/paper/2208.03895/) (2022)
- [Scaling Session-Based Transformer Recommendations using Optimized Negative Sampling and Loss Functions](https://mlsanity.com/paper/2307.14906/) (2023)
- [RECE: Reduced Cross-Entropy Loss for Large-Catalogue Sequential Recommenders](https://mlsanity.com/paper/2408.02354/) (2024)
- [Fast Amortized Inference and Learning in Log-linear Models with Randomly Perturbed Nearest Neighbor Search](https://mlsanity.com/paper/1707.03372/) (2017)
- [Deep Networks With Large Output Spaces](https://mlsanity.com/paper/1412.7479/) (2014)
- [Tensor-based Sequential Learning via Hankel Matrix Representation for Next Item Recommendations](https://mlsanity.com/paper/2212.05720/) (2022)

## Built on by (most cited first, 15 of 15)

- [Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders](https://mlsanity.com/paper/2507.16289/) (2025)
- [eSASRec: Enhancing Transformer-based Recommendations in a Modular Fashion](https://mlsanity.com/paper/2508.06450/) (2025)
- [Sparse Autoencoders for Sequential Recommendation Models: Interpretation and Flexible Control](https://mlsanity.com/paper/2507.12202/) (2025)
- [Recommendation Is a Dish Better Served Warm](https://mlsanity.com/paper/2508.07856/) (2025)
- [Ultra Fast Warm Start Solution for Graph Recommendations](https://mlsanity.com/paper/2509.01549/) (2025)
- [Maximum Impact with Fewer Features: Efficient Feature Selection for Cold-Start Recommenders through Collaborative Importance Weighting](https://mlsanity.com/paper/2508.06455/) (2025)
- [Relative-Based Scaling Law for Neural Language Models](https://mlsanity.com/paper/2510.20387/) (2025)
- [Looking around you: external information enhances representations for event sequences](https://mlsanity.com/paper/2502.10205/) (2025)
- [How to model Human Actions distribution with Event Sequence Data](https://mlsanity.com/paper/2510.05856/) (2025)
- [SplitLight: An Exploratory Toolkit for Recommender Systems Datasets and Splits](https://mlsanity.com/paper/2602.19339/) (2026)
- [Position-Aware Sequential Attention for Accurate Next Item Recommendations](https://mlsanity.com/paper/2602.21052/) (2026)
- [Cross-Representation Knowledge Transfer for Improved Sequential Recommendations](https://mlsanity.com/paper/2602.23471/) (2026)
- [Bradley-Terry Rankings for Recommender Systems Across Dataset Taxonomies](https://mlsanity.com/paper/2606.07492/) (2026)
- [Batch Size or Negatives? A Selection Rule for Memory-Constrained Recommender Training](https://mlsanity.com/paper/2608.11061/) (2026)
- [Keeping the Index Open: The Recommendation-Side Cost of Shared Search and Recommendation](https://mlsanity.com/paper/2608.24079/) (2026)

## Abstract (arXiv)

Scalability issue plays a crucial role in productionizing modern recommender systems. Even lightweight architectures may suffer from high computational overload due to intermediate calculations, limiting their practicality in real-world applications. Specifically, applying full Cross-Entropy (CE) loss often yields state-of-the-art performance in terms of recommendations quality. Still, it suffers from excessive GPU memory utilization when dealing with large item catalogs. This paper introduces a novel Scalable Cross-Entropy (SCE) loss function in the sequential learning setup. It approximates the CE loss for datasets with large-size catalogs, enhancing both time efficiency and memory usage without compromising recommendations quality. Unlike traditional negative sampling methods, our approach utilizes a selective GPU-efficient computation strategy, focusing on the most informative elements of the catalog, particularly those most likely to be false positives. This is achieved by approximating the softmax distribution over a subset of the model outputs through the maximum inner product search. Experimental results on multiple datasets demonstrate the effectiveness of SCE in reducing peak memory usage by a factor of up to 100 compared to the alternatives, retaining or even exceeding their metrics values. The proposed approach also opens new perspectives for large-scale developments in different domains, such as large language models.