# Lightweight Self-Attentive Sequential Recommendation

arXiv: https://arxiv.org/abs/2108.11333 · HTML: https://mlsanity.com/paper/2108.11333/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 55 references that are in the map.
- First posted on arXiv 25 Aug 2021.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding](https://mlsanity.com/paper/1809.07426/) (2018)
- [Predicting Dynamic Embedding Trajectory in Temporal Interaction Networks](https://mlsanity.com/paper/1908.01207/) (2019)
- [S^3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization](https://mlsanity.com/paper/2008.07873/) (2020)
- [Pay Less Attention with Lightweight and Dynamic Convolutions](https://mlsanity.com/paper/1901.10430/) (2019)
- [Fusing Similarity Models with Markov Chains for Sparse Sequential Recommendation](https://mlsanity.com/paper/1609.09152/) (2016)
- [Lite Transformer with Long-Short Range Attention](https://mlsanity.com/paper/2004.11886/) (2020)
- [Billion-scale Commodity Embedding for E-commerce Recommendation in Alibaba](https://mlsanity.com/paper/1803.02349/) (2018)
- [Translation-based Recommendation](https://mlsanity.com/paper/1707.02410/) (2017)
- [Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems](https://mlsanity.com/paper/1909.02107/) (2019)
- [RepeatNet: A Repeat Aware Neural Recommendation Machine for Session-based Recommendation](https://mlsanity.com/paper/1812.02646/) (2018)
- [Self-Supervised Reinforcement Learning for Recommender Systems](https://mlsanity.com/paper/2006.05779/) (2020)
- [Rethinking the Item Order in Session-based Recommendation with Graph Neural Networks](https://mlsanity.com/paper/1911.11942/) (2019)
- [Learnable Embedding Sizes for Recommender Systems](https://mlsanity.com/paper/2101.07577/) (2021)
- [Compressing Word Embeddings via Deep Compositional Code Learning](https://mlsanity.com/paper/1711.01068/) (2017)
- [SDM: Sequential Deep Matching Model for Online Large-scale Recommender System](https://mlsanity.com/paper/1909.00385/) (2019)
- [GAG: Global Attributed Graph Neural Network for Streaming Session-based Recommendation](https://mlsanity.com/paper/2007.02747/) (2020)
- [Discovering Collaborative Signals for Next POI Recommendation with Iterative Seq2Graph Augmentation](https://mlsanity.com/paper/2106.15814/) (2021)
- [Exploiting Cross-Session Information for Session-based Recommendation with Graph Neural Networks](https://mlsanity.com/paper/2107.00852/) (2021)
- [Learning Elastic Embeddings for Customizing On-Device Recommenders](https://mlsanity.com/paper/2106.02223/) (2021)
- [Sequence-Aware Factorization Machines for Temporal Predictive Analytics](https://mlsanity.com/paper/1911.02752/) (2019)
- [Learning K-way D-dimensional Discrete Codes for Compact Embedding Representations](https://mlsanity.com/paper/1806.09464/) (2018)
- [Context-Aware Attention-Based Data Augmentation for POI Recommendation](https://mlsanity.com/paper/2106.15984/) (2021)

## Built on by (most cited first, 22 of 22)

- [Intent Contrastive Learning for Sequential Recommendation](https://mlsanity.com/paper/2202.02519/) (2022)
- [A Comprehensive Survey on Trustworthy Recommender Systems](https://mlsanity.com/paper/2209.10117/) (2022)
- [Graph Condensation for Inductive Node Representation Learning](https://mlsanity.com/paper/2307.15967/) (2023)
- [Decoupled Side Information Fusion for Sequential Recommendation](https://mlsanity.com/paper/2204.11046/) (2022)
- [Turning Dross Into Gold Loss: is BERT4Rec really better than SASRec?](https://mlsanity.com/paper/2309.07602/) (2023)
- [A Systematic Review and Replicability Study of BERT4Rec for Sequential Recommendation](https://mlsanity.com/paper/2207.07483/) (2022)
- [MELT: Mutual Enhancement of Long-Tailed User and Item for Sequential Recommendation](https://mlsanity.com/paper/2304.08382/) (2023)
- [A Step Toward World Models: A Survey on Robotic Manipulation](https://mlsanity.com/paper/2511.02097/) (2025)
- [Experimental Analysis of Large-scale Learnable Vector Storage Compression](https://mlsanity.com/paper/2311.15578/) (2023)
- [End-to-end Learnable Clustering for Intent Learning in Recommendation](https://mlsanity.com/paper/2401.05975/) (2024)
- [Accelerating Scalable Graph Neural Network Inference with Node-Adaptive Propagation](https://mlsanity.com/paper/2310.10998/) (2023)
- [Determinantal Point Process Likelihoods for Sequential Recommendation](https://mlsanity.com/paper/2204.11562/) (2022)
- [Scaling Sequential Recommendation Models with Transformers](https://mlsanity.com/paper/2412.07585/) (2024)
- [Adversarial Item Promotion on Visually-Aware Recommender Systems by Guided Diffusion](https://mlsanity.com/paper/2312.15826/) (2023)
- [Exploiting Session Information in BERT-based Session-aware Sequential Recommendation](https://mlsanity.com/paper/2204.10851/) (2022)
- [ELECRec: Training Sequential Recommenders as Discriminators](https://mlsanity.com/paper/2204.02011/) (2022)
- [Generating Negative Samples for Sequential Recommendation](https://mlsanity.com/paper/2208.03645/) (2022)
- [eSASRec: Enhancing Transformer-based Recommendations in a Modular Fashion](https://mlsanity.com/paper/2508.06450/) (2025)
- [Beyond Learning from Next Item: Sequential Recommendation via Personalized Interest Sustainability](https://mlsanity.com/paper/2209.06644/) (2022)
- [Proxy-based Item Representation for Attribute and Context-aware Recommendation](https://mlsanity.com/paper/2312.06145/) (2023)
- [Towards Lightweight Cross-domain Sequential Recommendation via External Attention-enhanced Graph Convolution Network](https://mlsanity.com/paper/2302.03221/) (2023)
- [Lightweight yet Efficient: An External Attentive Graph Convolutional Network with Positional Prompts for Sequential Recommendation](https://mlsanity.com/paper/2502.15331/) (2025)

## Abstract (arXiv)

Modern deep neural networks (DNNs) have greatly facilitated the development of sequential recommender systems by achieving state-of-the-art recommendation performance on various sequential recommendation tasks. Given a sequence of interacted items, existing DNN-based sequential recommenders commonly embed each item into a unique vector to support subsequent computations of the user interest. However, due to the potentially large number of items, the over-parameterised item embedding matrix of a sequential recommender has become a memory bottleneck for efficient deployment in resource-constrained environments, e.g., smartphones and other edge devices. Furthermore, we observe that the widely-used multi-head self-attention, though being effective in modelling sequential dependencies among items, heavily relies on redundant attention units to fully capture both global and local item-item transition patterns within a sequence. In this paper, we introduce a novel lightweight self-attentive network (LSAN) for sequential recommendation. To aggressively compress the original embedding matrix, LSAN leverages the notion of compositional embeddings, where each item embedding is composed by merging a group of selected base embedding vectors derived from substantially smaller embedding matrices. Meanwhile, to account for the intrinsic dynamics of each item, we further propose a temporal context-aware embedding composition scheme. Besides, we develop an innovative twin-attention network that alleviates the redundancy of the traditional multi-head self-attention while retaining full capacity for capturing long- and short-term (i.e., global and local) item dependencies. Comprehensive experiments demonstrate that LSAN significantly advances the accuracy and memory efficiency of existing sequential recommenders.