# Empowering Large Language Model for Sequential Recommendation via Multimodal Embeddings and Semantic IDs

arXiv: https://arxiv.org/abs/2509.02017 · HTML: https://mlsanity.com/paper/2509.02017/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 65 references that are in the map.
- First posted on arXiv 2 Sep 2025.

## Builds on (most cited first, 30 of 36)

- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Learning Transferable Features with Deep Adaptation Networks](https://mlsanity.com/paper/1502.02791/) (2015)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [Image-based Recommendations on Styles and Substitutes](https://mlsanity.com/paper/1506.04757/) (2015)
- [Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks](https://mlsanity.com/paper/2208.10442/) (2022)
- [TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation](https://mlsanity.com/paper/2305.00447/) (2023)
- [Recommender Systems with Generative Retrieval](https://mlsanity.com/paper/2305.05065/) (2023)
- [Adapting Large Language Models by Integrating Collaborative Semantics for Recommendation](https://mlsanity.com/paper/2311.09049/) (2023)
- [Where to Go Next for Recommender Systems? ID- vs. Modality-based Recommender Models Revisited](https://mlsanity.com/paper/2303.13835/) (2023)
- [Learnable Item Tokenization for Generative Recommendation](https://mlsanity.com/paper/2405.07314/) (2024)
- [Deep Reinforcement Learning for Page-wise Recommendations](https://mlsanity.com/paper/1805.02343/) (2018)
- [Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning](https://mlsanity.com/paper/1802.06501/) (2018)
- [LLaRA: Large Language-Recommendation Assistant](https://mlsanity.com/paper/2312.02445/) (2023)
- [LLM-ESR: Large Language Models Enhancement for Long-tailed Sequential Recommendation](https://mlsanity.com/paper/2405.20646/) (2024)
- [Multimodal Recommender Systems: A Survey](https://mlsanity.com/paper/2302.03883/) (2023)
- [M3oE: Multi-Domain Multi-Task Mixture-of Experts Recommendation Framework](https://mlsanity.com/paper/2404.18465/) (2024)
- [Multi-Task Deep Recommender Systems: A Survey](https://mlsanity.com/paper/2302.03525/) (2023)
- [KuaiSim: A Comprehensive Simulator for Recommender Systems](https://mlsanity.com/paper/2309.12645/) (2023)
- [On the Embedding Collapse when Scaling up Recommendation Models](https://mlsanity.com/paper/2310.04400/) (2023)
- [Rethinking Large Language Model Architectures for Sequential Recommendations](https://mlsanity.com/paper/2402.09543/) (2024)
- [AutoDenoise: Automatic Data Instance Denoising for Recommendations](https://mlsanity.com/paper/2303.06611/) (2023)
- [HAMUR: Hyper Adapter for Multi-Domain Recommendation](https://mlsanity.com/paper/2309.06217/) (2023)
- [AlignRec: Aligning and Training in Multimodal Recommendations](https://mlsanity.com/paper/2403.12384/) (2024)
- [LinRec: Linear Attention Mechanism for Long-term Sequential Recommender Systems](https://mlsanity.com/paper/2411.01537/) (2024)
- [STEM: Unleashing the Power of Embeddings for Multi-task Recommendation](https://mlsanity.com/paper/2308.13537/) (2023)
- [LLM4MSR: An LLM-Enhanced Paradigm for Multi-Scenario Recommendation](https://mlsanity.com/paper/2406.12529/) (2024)
- [LLMEmb: Large Language Model Can Be a Good Embedding Generator for Sequential Recommendation](https://mlsanity.com/paper/2409.19925/) (2024)
- [Exploration and Regularization of the Latent Action Space in Recommendation](https://mlsanity.com/paper/2302.03431/) (2023)
- [Content-Based Collaborative Generation for Recommender Systems](https://mlsanity.com/paper/2403.18480/) (2024)
- [Bridge the Domains: Large Language Models Enhanced Cross-domain Sequential Recommendation](https://mlsanity.com/paper/2504.18383/) (2025)

## Built on by (most cited first, 2 of 2)

- [Emotion and Intention Guided Multi-Modal Learning for Sticker Response Selection](https://mlsanity.com/paper/2511.17587/) (2025)
- [Fusion and Alignment Enhancement with Large Language Models for Tail-item Sequential Recommendation](https://mlsanity.com/paper/2604.03688/) (2026)

## Abstract (arXiv)

Sequential recommendation (SR) aims to capture users' dynamic interests and sequential patterns based on their historical interactions. Recently, the powerful capabilities of large language models (LLMs) have driven their adoption in SR. However, we identify two critical challenges in existing LLM-based SR methods: 1) embedding collapse when incorporating pre-trained collaborative embeddings and 2) catastrophic forgetting of quantized embeddings when utilizing semantic IDs. These issues dampen the model scalability and lead to suboptimal recommendation performance. Therefore, based on LLMs like Llama3-8B-instruct, we introduce a novel SR framework named MME-SID, which integrates multimodal embeddings and quantized embeddings to mitigate embedding collapse. Additionally, we propose a Multimodal Residual Quantized Variational Autoencoder (MM-RQ-VAE) with maximum mean discrepancy as the reconstruction loss and contrastive learning for alignment, which effectively preserve intra-modal distance information and capture inter-modal correlations, respectively. To further alleviate catastrophic forgetting, we initialize the model with the trained multimodal code embeddings. Finally, we fine-tune the LLM efficiently using LoRA in a multimodal frequency-aware fusion manner. Extensive experiments on three public datasets validate the superior performance of MME-SID thanks to its capability to mitigate embedding collapse and catastrophic forgetting. The implementation code and datasets are publicly available for reproduction: https://github.com/Applied-Machine-Learning-Lab/MME-SID.