# Position-Aware Sequential Attention for Accurate Next Item Recommendations

arXiv: https://arxiv.org/abs/2602.21052 · HTML: https://mlsanity.com/paper/2602.21052/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 26 references that are in the map.
- First posted on arXiv 24 Feb 2026.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [Rethinking Positional Encoding in Language Pre-training](https://mlsanity.com/paper/2006.15595/) (2020)
- [Yelp Dataset Challenge: Review Rating Prediction](https://mlsanity.com/paper/1605.05362/) (2016)
- [Turning Dross Into Gold Loss: is BERT4Rec really better than SASRec?](https://mlsanity.com/paper/2309.07602/) (2023)
- [MEANTIME: Mixture of Attention Mechanisms with Multi-temporal Embeddings for Sequential Recommendation](https://mlsanity.com/paper/2008.08273/) (2020)
- [Scalable Cross-Entropy Loss for Sequential Recommendations with Large Item Catalogs](https://mlsanity.com/paper/2409.18721/) (2024)
- [From Variability to Stability: Advancing RecSys Benchmarking Practices](https://mlsanity.com/paper/2402.09766/) (2024)
- [Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders](https://mlsanity.com/paper/2507.16289/) (2025)
- [RECE: Reduced Cross-Entropy Loss for Large-Catalogue Sequential Recommenders](https://mlsanity.com/paper/2408.02354/) (2024)
- [Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion](https://mlsanity.com/paper/2310.02012/) (2023)
- [Yambda-5B -- A Large-Scale Multi-modal Dataset for Ranking And Retrieval](https://mlsanity.com/paper/2505.22238/) (2025)
- [Positional encoding is not the same as context: A study on positional encoding for sequential recommendation](https://mlsanity.com/paper/2405.10436/) (2024)
- [Rotate Both Ways: Time-and-Order RoPE for Generative Recommendation](https://mlsanity.com/paper/2510.20455/) (2025)
- [Backward Gradient Normalization in Deep Neural Networks](https://mlsanity.com/paper/2106.09475/) (2021)
- [A Contextual-Aware Position Encoding for Sequential Recommendation](https://mlsanity.com/paper/2502.09027/) (2025)

## Abstract (arXiv)

Sequential self-attention models usually rely on additive positional embeddings, which inject positional information into item representations at the input. In the absence of positional signals, the attention block is permutation-equivariant over sequence positions and thus has no intrinsic notion of temporal order beyond causal masking. We argue that additive positional embeddings make the attention mechanism only superficially sensitive to sequence order: positional information is entangled with item embedding semantics, propagates weakly in deep architectures, and limits the ability to capture rich sequential patterns. To address these limitations, we introduce a kernelized self-attention mechanism, where a learnable positional kernel operates purely in the position space, disentangled from semantic similarity, and directly modulates attention weights. When applied per attention block, this kernel enables adaptive multi-scale sequential modeling. Experiments on standard next-item prediction benchmarks show that our positional kernel attention consistently improves over strong competing baselines.