# MEANTIME: Mixture of Attention Mechanisms with Multi-temporal Embeddings for Sequential Recommendation

arXiv: https://arxiv.org/abs/2008.08273 · HTML: https://mlsanity.com/paper/2008.08273/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Aug 2020.

## Built on by (most cited first, 17 of 17)

- [Improving Transformers with Probabilistic Attention Keys](https://mlsanity.com/paper/2110.08678/) (2021)
- [A Systematic Review and Replicability Study of BERT4Rec for Sequential Recommendation](https://mlsanity.com/paper/2207.07483/) (2022)
- [Hierarchical Time-Aware Mixture of Experts for Multi-Modal Sequential Recommendation](https://mlsanity.com/paper/2501.14269/) (2025)
- [Effective and Efficient Training for Sequential Recommendation using Recency Sampling](https://mlsanity.com/paper/2207.02643/) (2022)
- [Attention Mixtures for Time-Aware Sequential Recommendation](https://mlsanity.com/paper/2304.08158/) (2023)
- [Autoregressive Generation Strategies for Top-K Sequential Recommendations](https://mlsanity.com/paper/2409.17730/) (2024)
- [Exploiting Session Information in BERT-based Session-aware Sequential Recommendation](https://mlsanity.com/paper/2204.10851/) (2022)
- [UniRec: A Dual Enhancement of Uniformity and Frequency in Sequential Recommendations](https://mlsanity.com/paper/2406.18470/) (2024)
- [VA-AR: Learning Velocity-Aware Action Representations with Mixture of Window Attention](https://mlsanity.com/paper/2503.11004/) (2025)
- [Towards a General Pre-training Framework for Adaptive Learning in MOOCs](https://mlsanity.com/paper/2208.04708/) (2022)
- [Enhancing Time Awareness in Generative Recommendation](https://mlsanity.com/paper/2509.13957/) (2025)
- [CPMR: Context-Aware Incremental Sequential Recommendation with Pseudo-Multi-Task Learning](https://mlsanity.com/paper/2309.04802/) (2023)
- [Sequential Recommendation via Adaptive Robust Attention with Multi-dimensional Embeddings](https://mlsanity.com/paper/2409.05022/) (2024)
- [Sequential Recommendation on Temporal Proximities with Contrastive Learning and Self-Attention](https://mlsanity.com/paper/2402.09784/) (2024)
- [Position-Aware Sequential Attention for Accurate Next Item Recommendations](https://mlsanity.com/paper/2602.21052/) (2026)
- [CAMMSR: Category-Guided Attentive Mixture of Experts for Multimodal Sequential Recommendation](https://mlsanity.com/paper/2603.04320/) (2026)
- [Zero-Observation User Reactivation with Gap-Driven Dimensional Gating](https://mlsanity.com/paper/2607.19802/) (2026)

## Abstract (arXiv)

Recently, self-attention based models have achieved state-of-the-art performance in sequential recommendation task. Following the custom from language processing, most of these models rely on a simple positional embedding to exploit the sequential nature of the user's history. However, there are some limitations regarding the current approaches. First, sequential recommendation is different from language processing in that timestamp information is available. Previous models have not made good use of it to extract additional contextual information. Second, using a simple embedding scheme can lead to information bottleneck since the same embedding has to represent all possible contextual biases. Third, since previous models use the same positional embedding in each attention head, they can wastefully learn overlapping patterns. To address these limitations, we propose MEANTIME (MixturE of AtteNTIon mechanisms with Multi-temporal Embeddings) which employs multiple types of temporal embeddings designed to capture various patterns from the user's behavior sequence, and an attention structure that fully leverages such diversity. Experiments on real-world data show that our proposed method outperforms current state-of-the-art sequential recommendation methods, and we provide an extensive ablation study to analyze how the model gains from the diverse positional information.