# Modeling Sequences as Distributions with Uncertainty for Sequential Recommendation

arXiv: https://arxiv.org/abs/2106.06165 · HTML: https://mlsanity.com/paper/2106.06165/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 37 references that are in the map.
- First posted on arXiv 11 Jun 2021.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation](https://mlsanity.com/paper/2002.02126/) (2020)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [Image-based Recommendations on Styles and Substitutes](https://mlsanity.com/paper/1506.04757/) (2015)
- [Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding](https://mlsanity.com/paper/1809.07426/) (2018)
- [Deep Gaussian Embedding of Graphs: Unsupervised Inductive Learning via Ranking](https://mlsanity.com/paper/1707.03815/) (2017)
- [Hierarchical Gating Networks for Sequential Recommendation](https://mlsanity.com/paper/1906.09217/) (2019)
- [Word Representations via Gaussian Embedding](https://mlsanity.com/paper/1412.6623/) (2014)
- [Translation-based Recommendation](https://mlsanity.com/paper/1707.02410/) (2017)
- [Modeling Uncertainty with Hedged Instance Embedding](https://mlsanity.com/paper/1810.00319/) (2018)
- [Augmenting Sequential Recommendation with Pseudo-Prior Items via Reversely Pre-training Transformer](https://mlsanity.com/paper/2105.00522/) (2021)
- [Visually-Aware Fashion Recommendation and Design with Generative Image Models](https://mlsanity.com/paper/1711.02231/) (2017)
- [Latent Relational Metric Learning via Memory-based Attention for Collaborative Ranking](https://mlsanity.com/paper/1707.05176/) (2017)
- [CosRec: 2D Convolutional Neural Networks for Sequential Recommendation](https://mlsanity.com/paper/1908.09972/) (2019)
- [Are Pretrained Transformers Robust in Intent Classification? A Missing Ingredient in Evaluation of Out-of-Scope Intent Detection](https://mlsanity.com/paper/2106.04564/) (2021)
- [BasConv: Aggregating Heterogeneous Interactions for Basket Recommendation with Graph Convolutional Neural Network](https://mlsanity.com/paper/2001.09900/) (2020)
- [Basket Recommendation with Multi-Intent Translation Graph Neural Network](https://mlsanity.com/paper/2010.11419/) (2020)
- [Enriching Non-Autoregressive Transformer with Syntactic and SemanticStructures for Neural Machine Translation](https://mlsanity.com/paper/2101.08942/) (2021)
- [Pseudo Siamese Network for Few-shot Intent Generation](https://mlsanity.com/paper/2105.00896/) (2021)
- [M2: Mixed Models with Preferences, Popularities and Transitions for Next-Basket Recommendation](https://mlsanity.com/paper/2004.01646/) (2020)

## Built on by (most cited first, 22 of 22)

- [DiffuRec: A Diffusion Model for Sequential Recommendation](https://mlsanity.com/paper/2304.00686/) (2023)
- [Continuous-Time Sequential Recommendation with Temporal Graph Collaborative Transformer](https://mlsanity.com/paper/2108.06625/) (2021)
- [Contrastive Self-supervised Sequential Recommendation with Robust Augmentation](https://mlsanity.com/paper/2108.06479/) (2021)
- [Sequential Recommendation via Stochastic Self-Attention](https://mlsanity.com/paper/2201.06035/) (2022)
- [A Survey on Diffusion Models for Recommender Systems](https://mlsanity.com/paper/2409.05033/) (2024)
- [DSKReG: Differentiable Sampling on Knowledge Graph for Recommendation with Relational GNN](https://mlsanity.com/paper/2108.11883/) (2021)
- [ContrastVAE: Contrastive Variational AutoEncoder for Sequential Recommendation](https://mlsanity.com/paper/2209.00456/) (2022)
- [MELT: Mutual Enhancement of Long-Tailed User and Item for Sequential Recommendation](https://mlsanity.com/paper/2304.08382/) (2023)
- [Uncertainty Quantification and Decomposition for LLM-based Recommendation](https://mlsanity.com/paper/2501.17630/) (2025)
- [Rethinking Missing Data: Aleatoric Uncertainty-Aware Recommendation](https://mlsanity.com/paper/2209.11679/) (2022)
- [NR4DER: Neural Re-ranking for Diversified Exercise Recommendation](https://mlsanity.com/paper/2506.06341/) (2025)
- [Sequential Recommendation with Auxiliary Item Relationships via Multi-Relational Transformer](https://mlsanity.com/paper/2210.13572/) (2022)
- [Episodes Discovery Recommendation with Multi-Source Augmentations](https://mlsanity.com/paper/2301.01737/) (2023)
- [Mutual Wasserstein Discrepancy Minimization for Sequential Recommendation](https://mlsanity.com/paper/2301.12197/) (2023)
- [Sequential Recommendation with Probabilistic Logical Reasoning](https://mlsanity.com/paper/2304.11383/) (2023)
- [Sequential Recommendation with Controllable Diversification: Representation Degeneration and Diversity](https://mlsanity.com/paper/2306.11986/) (2023)
- [Alleviating User-Sensitive bias with Fair Generative Sequential Recommendation Model](https://mlsanity.com/paper/2506.19777/) (2025)
- [G-STO: Sequential Main Shopping Intention Detection via Graph-Regularized Stochastic Transformer](https://mlsanity.com/paper/2306.14314/) (2023)
- [Enhancing Recommendation with Denoising Auxiliary Task](https://mlsanity.com/paper/2409.17402/) (2024)
- [Heterogeneous Sequel-Aware Graph Neural Networks for Sequential Learning](https://mlsanity.com/paper/2506.05625/) (2025)
- [Multi-Item-Query Attention for Stable Sequential Recommendation](https://mlsanity.com/paper/2509.24424/) (2025)
- [Knowledge Graphs Meet Graph Neural Networks: A Comprehensive Survey](https://mlsanity.com/paper/2607.09666/) (2026)

## Abstract (arXiv)

The sequential patterns within the user interactions are pivotal for representing the user's preference and capturing latent relationships among items. The recent advancements of sequence modeling by Transformers advocate the community to devise more effective encoders for the sequential recommendation. Most existing sequential methods assume users are deterministic. However, item-item transitions might fluctuate significantly in several item aspects and exhibit randomness of user interests. This textitstochastic characteristics brings up a solid demand to include uncertainties in representing sequences and items. Additionally, modeling sequences and items with uncertainties expands users' and items' interaction spaces, thus further alleviating cold-start problems. In this work, we propose a Distribution-based Transformer for Sequential Recommendation (DT4SR), which injects uncertainties into sequential modeling. We use Elliptical Gaussian distributions to describe items and sequences with uncertainty. We describe the uncertainty in items and sequences as Elliptical Gaussian distribution. And we adopt Wasserstein distance to measure the similarity between distributions. We devise two novel Trans-formers for modeling mean and covariance, which guarantees the positive-definite property of distributions. The proposed method significantly outperforms the state-of-the-art methods. The experiments on three benchmark datasets also demonstrate its effectiveness in alleviating cold-start issues. The code is available inhttps://github.com/DyGRec/DT4SR.