# An Attentive Inductive Bias for Sequential Recommendation beyond the Self-Attention

arXiv: https://arxiv.org/abs/2312.10325 · HTML: https://mlsanity.com/paper/2312.10325/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 48 references that are in the map.
- First posted on arXiv 16 Dec 2023.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation](https://mlsanity.com/paper/2002.02126/) (2020)
- [Graph Convolutional Neural Networks for Web-Scale Recommender Systems](https://mlsanity.com/paper/1806.01973/) (2018)
- [Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning](https://mlsanity.com/paper/1801.07606/) (2018)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [Image-based Recommendations on Styles and Substitutes](https://mlsanity.com/paper/1506.04757/) (2015)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [Graph Neural Networks in Recommender Systems: A Survey](https://mlsanity.com/paper/2011.02260/) (2020)
- [Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding](https://mlsanity.com/paper/1809.07426/) (2018)
- [Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth](https://mlsanity.com/paper/2103.03404/) (2021)
- [S^3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization](https://mlsanity.com/paper/2008.07873/) (2020)
- [DeepViT: Towards Deeper Vision Transformer](https://mlsanity.com/paper/2103.11886/) (2021)
- [Fusing Similarity Models with Markov Chains for Sparse Sequential Recommendation](https://mlsanity.com/paper/1609.09152/) (2016)
- [Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice](https://mlsanity.com/paper/2203.05962/) (2022)
- [Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation](https://mlsanity.com/paper/2110.05730/) (2021)
- [Filter-enhanced MLP is All You Need for Sequential Recommendation](https://mlsanity.com/paper/2202.13556/) (2022)
- [Discrete Signal Processing on Graphs: Frequency Analysis](https://mlsanity.com/paper/1307.0468/) (2013)
- [GREAD: Graph Neural Reaction-Diffusion Networks](https://mlsanity.com/paper/2211.14208/) (2022)
- [Frequency Enhanced Hybrid Attention Network for Sequential Recommendation](https://mlsanity.com/paper/2304.09184/) (2023)
- [Vision Transformers with Patch Diversification](https://mlsanity.com/paper/2104.12753/) (2021)
- [Current Challenges and Visions in Music Recommender Systems Research](https://mlsanity.com/paper/1710.03208/) (2017)
- [ContraNorm: A Contrastive Learning Perspective on Oversmoothing and Beyond](https://mlsanity.com/paper/2303.06562/) (2023)
- [Blurring-Sharpening Process Models for Collaborative Filtering](https://mlsanity.com/paper/2211.09324/) (2022)
- [Linear Recurrent Units for Sequential Recommendation](https://mlsanity.com/paper/2310.02367/) (2023)
- [LT-OCF: Learnable-Time ODE-based Collaborative Filtering](https://mlsanity.com/paper/2108.06208/) (2021)
- [Linear, or Non-Linear, That is the Question!](https://mlsanity.com/paper/2111.07265/) (2021)
- [Attention Calibration for Transformer-based Sequential Recommendation](https://mlsanity.com/paper/2308.09419/) (2023)
- [Mixed Attention Network for Cross-domain Sequential Recommendation](https://mlsanity.com/paper/2311.08272/) (2023)

## Built on by (most cited first, 28 of 28)

- [Structured Spectral Reasoning for Frequency-Adaptive Multimodal Recommendation](https://mlsanity.com/paper/2512.01372/) (2025)
- [SCONE: A Novel Stochastic Sampling to Generate Contrastive Views and Hard Negative Samples for Recommendation](https://mlsanity.com/paper/2405.00287/) (2024)
- [SVD-AE: Simple Autoencoders for Collaborative Filtering](https://mlsanity.com/paper/2405.04746/) (2024)
- [CSRec: Rethinking Sequential Recommendation from A Causal Perspective](https://mlsanity.com/paper/2409.05872/) (2024)
- [Temporal Linear Item-Item Model for Sequential Recommendation](https://mlsanity.com/paper/2412.07382/) (2024)
- [Learning Advanced Self-Attention for Linear Transformers in the Singular Value Domain](https://mlsanity.com/paper/2505.08516/) (2025)
- [RDGCL: Reaction-Diffusion Graph Contrastive Learning for Recommendation](https://mlsanity.com/paper/2312.16563/) (2023)
- [Exploiting Inter-Session Information with Frequency-enhanced Dual-Path Networks for Sequential Recommendation](https://mlsanity.com/paper/2511.06285/) (2025)
- [Enhancing ID and Text Fusion via Alternative Training in Session-based Recommendation](https://mlsanity.com/paper/2402.08921/) (2024)
- [Lightweight yet Fine-grained: A Graph Capsule Convolutional Network with Subspace Alignment for Shared-account Sequential Recommendation](https://mlsanity.com/paper/2412.13408/) (2024)
- [Spatial-Temporal Knowledge Distillation for Takeaway Recommendation](https://mlsanity.com/paper/2412.16502/) (2024)
- [Compute Only Once: UG-Separation for Efficient Large Recommendation Models](https://mlsanity.com/paper/2602.10455/) (2026)
- [A novel forecasting framework combining virtual samples and enhanced Transformer models for tourism demand forecasting](https://mlsanity.com/paper/2503.19423/) (2025)
- [Multi-Perspective Attention Mechanism for Bias-Aware Sequential Recommendation](https://mlsanity.com/paper/2504.05323/) (2025)
- [High-Fidelity Differential-information Driven Binary Vision Transformer](https://mlsanity.com/paper/2507.02222/) (2025)
- [Beyond Semantic Understanding: Preserving Collaborative Frequency Components in LLM-based Recommendation](https://mlsanity.com/paper/2508.10312/) (2025)
- [Filter then Attend: Improving attention-based Time Series Forecasting with Spectral Filtering](https://mlsanity.com/paper/2508.20206/) (2025)
- [CausalRec: A CausalBoost Attention Model for Sequential Recommendation](https://mlsanity.com/paper/2510.21333/) (2025)
- [TV-Rec: Time-Variant Convolutional Filter for Sequential Recommendation](https://mlsanity.com/paper/2510.25259/) (2025)
- [A Systematic Reproducibility Study of BSARec for Sequential Recommendation](https://mlsanity.com/paper/2512.17442/) (2025)
- [Multimodal Enhancement of Sequential Recommendation](https://mlsanity.com/paper/2602.07207/) (2026)
- [Sequences as Nodes for Contrastive Multimodal Graph Recommendation](https://mlsanity.com/paper/2602.07208/) (2026)
- [Sensory-Aware Sequential Recommendation via Review-Distilled Representations](https://mlsanity.com/paper/2603.02709/) (2026)
- [DisenReason: Behavior Disentanglement and Latent Reasoning for Shared-Account Sequential Recommendation](https://mlsanity.com/paper/2603.03782/) (2026)
- [The Bandit's Blind Spot: The Critical Role of User State Representation in Recommender Systems](https://mlsanity.com/paper/2604.26651/) (2026)
- [FAiT: Frequency-Aware Inverted Transformer for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2606.01306/) (2026)
- [Residual Dominance as a Structural Account of Last-Item Reliance in Causal Self-Attention Recommenders](https://mlsanity.com/paper/2608.14021/) (2026)
- [RouteRec: Behavior-Guided Sparse Routing for Sequential Recommendation](https://mlsanity.com/paper/2609.39007/) (2026)

## Abstract (arXiv)

Sequential recommendation (SR) models based on Transformers have achieved remarkable successes. The self-attention mechanism of Transformers for computer vision and natural language processing suffers from the oversmoothing problem, i.e., hidden representations becoming similar to tokens. In the SR domain, we, for the first time, show that the same problem occurs. We present pioneering investigations that reveal the low-pass filtering nature of self-attention in the SR, which causes oversmoothing. To this end, we propose a novel method called Beyond Self-Attention for Sequential Recommendation (BSARec), which leverages the Fourier transform to i) inject an inductive bias by considering fine-grained sequential patterns and ii) integrate low and high-frequency information to mitigate oversmoothing. Our discovery shows significant advancements in the SR domain and is expected to bridge the gap for existing Transformer-based SR models. We test our proposed approach through extensive experiments on 6 benchmark datasets. The experimental results demonstrate that our model outperforms 7 baseline methods in terms of recommendation performance. Our code is available at https://github.com/yehjin-shin/BSARec.