# Facet-Aware Multi-Head Mixture-of-Experts Model for Sequential Recommendation

arXiv: https://arxiv.org/abs/2411.01457 · HTML: https://mlsanity.com/paper/2411.01457/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in MoE for recommender systems, one of 12 directions in Mixture-of-experts.
- Builds on 29 of its 41 references that are in the map.
- First posted on arXiv 3 Nov 2024.

## Placed in

- Mixture-of-experts / [MoE for recommender systems](https://mlsanity.com/moe/moe-for-recommender-systems/)

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://mlsanity.com/paper/2112.06905/) (2021)
- [Wide & Deep Learning for Recommender Systems](https://mlsanity.com/paper/1606.07792/) (2016)
- [Image-based Recommendations on Styles and Substitutes](https://mlsanity.com/paper/1506.04757/) (2015)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding](https://mlsanity.com/paper/1809.07426/) (2018)
- [Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)](https://mlsanity.com/paper/2203.13366/) (2022)
- [AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks](https://mlsanity.com/paper/1810.11921/) (2018)
- [Session-based Recommendation with Graph Neural Networks](https://mlsanity.com/paper/1811.00855/) (2018)
- [S^3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization](https://mlsanity.com/paper/2008.07873/) (2020)
- [On the Representation Collapse of Sparse Mixture of Experts](https://mlsanity.com/paper/2204.09179/) (2022)
- [Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation](https://mlsanity.com/paper/2110.05730/) (2021)
- [Sequential Recommender Systems: Challenges, Progress and Prospects](https://mlsanity.com/paper/2001.04830/) (2019)
- [A Survey on Session-based Recommender Systems](https://mlsanity.com/paper/1902.04864/) (2019)
- [Intent Contrastive Learning for Sequential Recommendation](https://mlsanity.com/paper/2202.02519/) (2022)
- [Sequential Recommendation with Graph Neural Networks](https://mlsanity.com/paper/2106.14226/) (2021)
- [A Survey on Trajectory Data Management, Analytics, and Learning](https://mlsanity.com/paper/2003.11547/) (2020)
- [Efficiently Leveraging Multi-level User Intent for Session-based Recommendation via Atten-Mixer Network](https://mlsanity.com/paper/2206.12781/) (2022)
- [Debiased Contrastive Learning for Sequential Recommendation](https://mlsanity.com/paper/2303.11780/) (2023)
- [CORE: Simple and Effective Session-based Recommendation within Consistent Representation Space](https://mlsanity.com/paper/2204.11067/) (2022)
- [Sequential Recommendation with Self-Attentive Multi-Adversarial Network](https://mlsanity.com/paper/2005.10602/) (2020)
- [LLM-Enhanced User-Item Interactions: Leveraging Edge Information for Optimized Recommendations](https://mlsanity.com/paper/2402.09617/) (2024)
- [Re4: Learning to Re-contrast, Re-attend, Re-construct for Multi-interest Recommendation](https://mlsanity.com/paper/2208.08011/) (2022)
- [Large Language Models as Data Augmenters for Cold-Start Item Recommendation](https://mlsanity.com/paper/2402.11724/) (2024)
- [SSLRec: A Self-Supervised Learning Framework for Recommendation](https://mlsanity.com/paper/2308.05697/) (2023)
- [Learning Multi-granularity User Intent Unit for Session-based Recommendation](https://mlsanity.com/paper/2112.13197/) (2021)

## Built on by (most cited first, 4 of 4)

- [Modeling Annotator Disagreement with Demographic-Aware Experts and Synthetic Perspectives](https://mlsanity.com/paper/2508.02853/) (2025)
- [Zenith: Scaling up Ranking Models for Billion-scale Livestreaming Recommendation](https://mlsanity.com/paper/2601.21285/) (2026)
- [Facet-Aware Multi-Head Mixture-of-Experts Model with Text-Enhanced Pre-training for Sequential Recommendation](https://mlsanity.com/paper/2601.12301/) (2026)
- [RouteRec: Behavior-Guided Sparse Routing for Sequential Recommendation](https://mlsanity.com/paper/2609.39007/) (2026)

## Abstract (arXiv)

Sequential recommendation (SR) systems excel at capturing users' dynamic preferences by leveraging their interaction histories. Most existing SR systems assign a single embedding vector to each item to represent its features, and various types of models are adopted to combine these item embeddings into a sequence representation vector to capture the user intent. However, we argue that this representation alone is insufficient to capture an item's multi-faceted nature (e.g., movie genres, starring actors). Besides, users often exhibit complex and varied preferences within these facets (e.g., liking both action and musical films in the facet of genre), which are challenging to fully represent. To address the issues above, we propose a novel structure called Facet-Aware Multi-Head Mixture-of-Experts Model for Sequential Recommendation (FAME). We leverage sub-embeddings from each head in the last multi-head attention layer to predict the next item separately. This approach captures the potential multi-faceted nature of items without increasing model complexity. A gating mechanism integrates recommendations from each head and dynamically determines their importance. Furthermore, we introduce a Mixture-of-Experts (MoE) network in each attention head to disentangle various user preferences within each facet. Each expert within the MoE focuses on a specific preference. A learnable router network is adopted to compute the importance weight for each expert and aggregate them. We conduct extensive experiments on four public sequential recommendation datasets and the results demonstrate the effectiveness of our method over existing baseline models.