# Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation

arXiv: https://arxiv.org/abs/2110.05730 · HTML: https://mlsanity.com/paper/2110.05730/

## Key facts

- Cited by 121 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Oct 2021.

## Built on by (most cited first, 30 of 121)

- [Generate What You Prefer: Reshaping Sequential Recommendation via Guided Diffusion](https://mlsanity.com/paper/2310.20453/) (2023)
- [E4SRec: An Elegant Effective Efficient Extensible Solution of Large Language Models for Sequential Recommendation](https://mlsanity.com/paper/2312.02443/) (2023)
- [Sparse Meets Dense: Unified Generative Recommendations with Cascaded Sparse-Dense Representations](https://mlsanity.com/paper/2503.02453/) (2025)
- [MISSRec: Pre-training and Transferring Multi-modal Interest-aware Sequence Representation for Recommendation](https://mlsanity.com/paper/2308.11175/) (2023)
- [Turning Dross Into Gold Loss: is BERT4Rec really better than SASRec?](https://mlsanity.com/paper/2309.07602/) (2023)
- [gSASRec: Reducing Overconfidence in Sequential Recommendation Trained with Negative Sampling](https://mlsanity.com/paper/2308.07192/) (2023)
- [An Attentive Inductive Bias for Sequential Recommendation beyond the Self-Attention](https://mlsanity.com/paper/2312.10325/) (2023)
- [Conditional Denoising Diffusion for Sequential Recommendation](https://mlsanity.com/paper/2304.11433/) (2023)
- [A Systematic Review and Replicability Study of BERT4Rec for Sequential Recommendation](https://mlsanity.com/paper/2207.07483/) (2022)
- [ContrastVAE: Contrastive Variational AutoEncoder for Sequential Recommendation](https://mlsanity.com/paper/2209.00456/) (2022)
- [Graph Masked Autoencoder for Sequential Recommendation](https://mlsanity.com/paper/2305.04619/) (2023)
- [SLMRec: Distilling Large Language Models into Small for Sequential Recommendation](https://mlsanity.com/paper/2405.17890/) (2024)
- [SSLRec: A Self-Supervised Learning Framework for Recommendation](https://mlsanity.com/paper/2308.05697/) (2023)
- [Uniform Sequence Better: Time Interval Aware Data Augmentation for Sequential Recommendation](https://mlsanity.com/paper/2212.08262/) (2022)
- [Knowledge Enhancement for Contrastive Multi-Behavior Recommendation](https://mlsanity.com/paper/2301.05403/) (2023)
- [Scalable Cross-Entropy Loss for Sequential Recommendations with Large Item Catalogs](https://mlsanity.com/paper/2409.18721/) (2024)
- [Hierarchical Time-Aware Mixture of Experts for Multi-Modal Sequential Recommendation](https://mlsanity.com/paper/2501.14269/) (2025)
- [Bridge the Domains: Large Language Models Enhanced Cross-domain Sequential Recommendation](https://mlsanity.com/paper/2504.18383/) (2025)
- [Effective and Efficient Training for Sequential Recommendation using Recency Sampling](https://mlsanity.com/paper/2207.02643/) (2022)
- [Distinguished Quantized Guidance for Diffusion-based Sequence Recommendation](https://mlsanity.com/paper/2501.17670/) (2025)
- [Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders](https://mlsanity.com/paper/2507.16289/) (2025)
- [FLIP: Fine-grained Alignment between ID-based Models and Pretrained Language Models for CTR Prediction](https://mlsanity.com/paper/2310.19453/) (2023)
- [Unveiling Vulnerabilities of Contrastive Recommender Systems to Poisoning Attacks](https://mlsanity.com/paper/2311.18244/) (2023)
- [RecDCL: Dual Contrastive Learning for Recommendation](https://mlsanity.com/paper/2401.15635/) (2024)
- [Text Matching Improves Sequential Recommendation by Reducing Popularity Biases](https://mlsanity.com/paper/2308.14029/) (2023)
- [APGL4SR: A Generic Framework with Adaptive and Personalized Global Collaborative Information in Sequential Recommendation](https://mlsanity.com/paper/2311.02816/) (2023)
- [End-to-end Learnable Clustering for Intent Learning in Recommendation](https://mlsanity.com/paper/2401.05975/) (2024)
- [LLM4DSR: Leveraging Large Language Model for Denoising Sequential Recommendation](https://mlsanity.com/paper/2408.08208/) (2024)
- [Sequential/Session-based Recommendations: Challenges, Approaches, Applications and Opportunities](https://mlsanity.com/paper/2205.10759/) (2022)
- [Diffusion-based Contrastive Learning for Sequential Recommendation](https://mlsanity.com/paper/2405.09369/) (2024)

## Abstract (arXiv)

Recent advancements of sequential deep learning models such as Transformer and BERT have significantly facilitated the sequential recommendation. However, according to our study, the distribution of item embeddings generated by these models tends to degenerate into an anisotropic shape, which may result in high semantic similarities among embeddings. In this paper, both empirical and theoretical investigations of this representation degeneration problem are first provided, based on which a novel recommender model DuoRec is proposed to improve the item embeddings distribution. Specifically, in light of the uniformity property of contrastive learning, a contrastive regularization is designed for DuoRec to reshape the distribution of sequence representations. Given the convention that the recommendation task is performed by measuring the similarity between sequence representations and item embeddings in the same space via dot product, the regularization can be implicitly applied to the item embedding distribution. Existing contrastive learning methods mainly rely on data level augmentation for user-item interaction sequences through item cropping, masking, or reordering and can hardly provide semantically consistent augmentation samples. In DuoRec, a model-level augmentation is proposed based on Dropout to enable better semantic preserving. Furthermore, a novel sampling strategy is developed, where sequences having the same target item are chosen hard positive samples. Extensive experiments conducted on five datasets demonstrate the superior performance of the proposed DuoRec model compared with baseline methods. Visualization results of the learned representations validate that DuoRec can largely alleviate the representation degeneration problem.