# Mixture of Sequence: Theme-Aware Mixture-of-Experts for Long-Sequence Recommendation

arXiv: https://arxiv.org/abs/2604.20858 · HTML: https://mlsanity.com/paper/2604.20858/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in MoE for recommender systems, one of 12 directions in Mixture-of-experts.
- Builds on 42 of its 160 references that are in the map.
- First posted on arXiv 1 Mar 2026.

## Placed in

- Mixture-of-experts / [MoE for recommender systems](https://mlsanity.com/moe/moe-for-recommender-systems/)

## Builds on (most cited first, 30 of 42)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2401.06066/) (2024)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://mlsanity.com/paper/1703.04247/) (2017)
- [LightGCL: Simple Yet Effective Graph Contrastive Learning for Recommendation](https://mlsanity.com/paper/2302.08191/) (2023)
- [Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs](https://mlsanity.com/paper/2510.17771/) (2025)
- [A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agents](https://mlsanity.com/paper/2601.12538/) (2026)
- [EE-Net: Exploitation-Exploration Neural Networks in Contextual Bandits](https://mlsanity.com/paper/2110.03177/) (2021)
- [How Smooth Is Attention?](https://mlsanity.com/paper/2312.14820/) (2023)
- [Deep Learning for Click-Through Rate Estimation](https://mlsanity.com/paper/2104.10584/) (2021)
- [Mamba4Rec: Towards Efficient Sequential Recommendation with Selective State Space Models](https://mlsanity.com/paper/2403.03900/) (2024)
- [iAgent: LLM Agent as a Shield between User and Recommender Systems](https://mlsanity.com/paper/2502.14662/) (2025)
- [Graphs Meet AI Agents: Taxonomy, Progress, and Future Opportunities](https://mlsanity.com/paper/2506.18019/) (2025)
- [Cold-Start Recommendation towards the Era of Large Language Models (LLMs): A Comprehensive Survey and Roadmap](https://mlsanity.com/paper/2501.01945/) (2025)
- [SLMRec: Distilling Large Language Models into Small for Sequential Recommendation](https://mlsanity.com/paper/2405.17890/) (2024)
- [Pave Your Own Path: Graph Gradual Domain Adaptation on Fused Gromov-Wasserstein Geodesics](https://mlsanity.com/paper/2505.12709/) (2025)
- [Graph4MM: Weaving Multimodal Learning with Structural Information](https://mlsanity.com/paper/2510.16990/) (2025)
- [COIN: Co-Cluster Infomax for Bipartite Graphs](https://mlsanity.com/paper/2206.00006/) (2022)
- [PyG-SSL: A Graph Self-Supervised Learning Toolkit](https://mlsanity.com/paper/2412.21151/) (2024)
- [Flow Matching Meets Biology and Life Science: A Survey](https://mlsanity.com/paper/2507.17731/) (2025)
- [CoFiRec: Coarse-to-Fine Tokenization for Generative Recommendation](https://mlsanity.com/paper/2511.22707/) (2025)
- [A Collaborative Ensemble Framework for CTR Prediction](https://mlsanity.com/paper/2411.13700/) (2024)
- [MORALISE: A Structured Benchmark for Moral Alignment in Visual Language Models](https://mlsanity.com/paper/2505.14728/) (2025)
- [PLANETALIGN: A Comprehensive Python Library for Benchmarking Network Alignment](https://mlsanity.com/paper/2505.21366/) (2025)
- [Hierarchical LoRA MoE for Efficient CTR Model Scaling](https://mlsanity.com/paper/2510.10432/) (2025)
- [Harnessing Consistency for Robust Test-Time LLM Ensemble](https://mlsanity.com/paper/2510.13855/) (2025)
- [CLIMB: Class-imbalanced Learning Benchmark on Tabular Data](https://mlsanity.com/paper/2505.17451/) (2025)

## Built on by (most cited first, 4 of 4)

- [Predict, Don't Iterate: Efficient Adaptive-Length Infilling for Diffusion Language Models](https://mlsanity.com/paper/2609.02108/) (2026)
- [UniMoMo: Expert Merging-Based MoE Acceleration for Large Recommendation Models](https://mlsanity.com/paper/2608.08627/) (2026)
- [ReMix: Reinforcement routing for mixtures of LoRAs in LLM finetuning](https://mlsanity.com/paper/2603.10160/) (2026)
- [One Model, Many Minds: Unlocking Multi-Agent Synergy in a Single Agent via Mixture of Roles](https://mlsanity.com/paper/2608.27338/) (2026)

## Abstract (arXiv)

Sequential recommendation has rapidly advanced in click-through rate prediction due to its ability to model dynamic user interests. A key challenge, however, lies in modeling long sequences: users often exhibit significant interest shifts, introducing substantial irrelevant or misleading information. Our empirical analysis corroborates this challenge and uncovers a recurring behavioral pattern in long sequences (textitsession hopping): user interests remain stable within short temporal spans (textitsessions) but shift drastically across sessions and may reappear after multiple sessions. To address this challenge, we propose the Mixture of Sequence (MoS) framework, a model-agnostic MoE approach that achieves accurate predictions by extracting theme-specific and multi-scale subsequences from noisy raw user sequences. First, MoS employs a theme-aware routing mechanism to adaptively learn the latent themes of user sequences and organizes these sequences into multiple coherent subsequences. Each subsequence contains only sessions aligned with a specific theme, thereby effectively filtering out irrelevant or even misleading information introduced by user interest shifts in session hopping. In addition, to alleviate potential information loss, we introduce a multi-scale fusion mechanism, which leverages three types of experts to capture global sequence characteristics, short-term user behaviors, and theme-specific semantic patterns. Together, these two mechanisms endow MoS with the ability to deliver accurate recommendations from multi-faceted and multi-scale perspectives. Experimental results demonstrate that MoS consistently achieves the SOTA performance while introducing fewer FLOPs compared with other MoE counterparts, providing strong evidence of its excellent balance between utility and efficiency. The code is available at https://github.com/xiaolin-cs/MoS.