# MISSRec: Pre-training and Transferring Multi-modal Interest-aware Sequence Representation for Recommendation

arXiv: https://arxiv.org/abs/2308.11175 · HTML: https://mlsanity.com/paper/2308.11175/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 97 references that are in the map.
- First posted on arXiv 22 Aug 2023.

## Builds on (most cited first, 30 of 54)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation](https://mlsanity.com/paper/2002.02126/) (2020)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [Towards a Unified View of Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2110.04366/) (2021)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding](https://mlsanity.com/paper/1809.07426/) (2018)
- [S^3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization](https://mlsanity.com/paper/2008.07873/) (2020)
- [Simple, Scalable Adaptation for Neural Machine Translation](https://mlsanity.com/paper/1909.08478/) (2019)
- [Fusing Similarity Models with Markov Chains for Sparse Sequential Recommendation](https://mlsanity.com/paper/1609.09152/) (2016)
- [VBPR: Visual Bayesian Personalized Ranking from Implicit Feedback](https://mlsanity.com/paper/1510.01784/) (2015)
- [Where to Go Next for Recommender Systems? ID- vs. Modality-based Recommender Models Revisited](https://mlsanity.com/paper/2303.13835/) (2023)
- [Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation](https://mlsanity.com/paper/2110.05730/) (2021)
- [Towards Universal Sequence Representation Learning for Recommender Systems](https://mlsanity.com/paper/2206.05941/) (2022)
- [A Simple Convolutional Generative Network for Next Item Recommendation](https://mlsanity.com/paper/1808.05163/) (2018)
- [Causal Intervention for Leveraging Popularity Bias in Recommendation](https://mlsanity.com/paper/2105.06067/) (2021)
- [Sequential Recommender Systems: Challenges, Progress and Prospects](https://mlsanity.com/paper/2001.04830/) (2019)
- [Multi-Interest Network with Dynamic Routing for Recommendation at Tmall](https://mlsanity.com/paper/1904.08030/) (2019)
- [Learning Vector-Quantized Item Representation for Transferable Sequential Recommenders](https://mlsanity.com/paper/2210.12316/) (2022)

## Built on by (most cited first, 30 of 31)

- [IISAN: Efficiently Adapting Multimodal Representation for Sequential Recommendation with Decoupled PEFT](https://mlsanity.com/paper/2404.02059/) (2024)
- [LEARN: Knowledge Adaptation from Large Language Model to Recommendation for Practical Industrial Application](https://mlsanity.com/paper/2405.03988/) (2024)
- [PMG : Personalized Multimodal Generation with Large Language Models](https://mlsanity.com/paper/2404.08677/) (2024)
- [Hierarchical Time-Aware Mixture of Experts for Multi-Modal Sequential Recommendation](https://mlsanity.com/paper/2501.14269/) (2025)
- [Multi-Modal Multi-Behavior Sequential Recommendation with Conditional Diffusion-Based Feature Denoising](https://mlsanity.com/paper/2508.05352/) (2025)
- [Multimodal Quantitative Language for Generative Recommendation](https://mlsanity.com/paper/2504.05314/) (2025)
- [Universal Item Tokenization for Transferable Generative Recommendation](https://mlsanity.com/paper/2504.04405/) (2025)
- [Semantic Codebook Learning for Dynamic Recommendation Models](https://mlsanity.com/paper/2408.00123/) (2024)
- [Captions Speak Louder than Images: Generalizing Foundation Models for E-commerce from High-quality Multimodal Instruction Data](https://mlsanity.com/paper/2410.17337/) (2024)
- [EvdCLIP: Improving Vision-Language Retrieval with Entity Visual Descriptions from Large Language Models](https://mlsanity.com/paper/2505.18594/) (2025)
- [MOON2.0: Dynamic Modality-balanced Multimodal Representation Learning for E-commerce Product Understanding](https://mlsanity.com/paper/2511.12449/) (2025)
- [MIM: Multi-modal Content Interest Modeling Paradigm for User Behavior Modeling](https://mlsanity.com/paper/2502.00321/) (2025)
- [Multi-Aspect Cross-modal Quantization for Generative Recommendation](https://mlsanity.com/paper/2511.15122/) (2025)
- [BiVRec: Bidirectional View-based Multimodal Sequential Recommendation](https://mlsanity.com/paper/2402.17334/) (2024)
- [RAT: Retrieval-Augmented Transformer for Click-Through Rate Prediction](https://mlsanity.com/paper/2404.02249/) (2024)
- [Federated Vision-Language-Recommendation with Personalized Fusion](https://mlsanity.com/paper/2410.08478/) (2024)
- [Catalog-Native LLM: Speaking Item-ID Dialect with Less Entanglement for Recommendation](https://mlsanity.com/paper/2510.05125/) (2025)
- [Learning Variable-Length Tokenization for Generative Recommendation](https://mlsanity.com/paper/2605.17779/) (2026)
- [SynGR: Unleashing the Potential of Cross-Modal Synergy for Generative Recommendation](https://mlsanity.com/paper/2605.18920/) (2026)
- [Bridging Domain Gaps between Pretrained Multimodal Models and Recommendations](https://mlsanity.com/paper/2502.15542/) (2025)
- [Intrinsic and Extrinsic Factor Disentanglement for Recommendation in Various Context Scenarios](https://mlsanity.com/paper/2503.03524/) (2025)
- [A Novel Mamba-based Sequential Recommendation Method](https://mlsanity.com/paper/2504.07398/) (2025)
- [Large-scale User Game Lifecycle Representation Learning](https://mlsanity.com/paper/2510.15412/) (2025)
- [Q-BERT4Rec: Quantized Semantic-ID Representation Learning for Multimodal Recommendation](https://mlsanity.com/paper/2512.02474/) (2025)
- [CAMMSR: Category-Guided Attentive Mixture of Experts for Multimodal Sequential Recommendation](https://mlsanity.com/paper/2603.04320/) (2026)
- [VLM2Rec: Resolving Modality Collapse in Vision-Language Model Embedders for Multimodal Sequential Recommendation](https://mlsanity.com/paper/2603.17450/) (2026)
- [Aligning Multimodal Sequential Recommendations via Robust Direct Preference Optimization with Sparse MoE](https://mlsanity.com/paper/2603.29259/) (2026)
- [TME-PSR: Time-aware, Multi-interest, and Explanation Personalization for Sequential Recommendation](https://mlsanity.com/paper/2604.09439/) (2026)
- [Sequential Modality Dropout for Robust Multi-Modal Sequential Recommendation](https://mlsanity.com/paper/2608.10240/) (2026)
- [Cross-Country Code-Mixing for Generative Recommendation](https://mlsanity.com/paper/2609.28972/) (2026)

## Abstract (arXiv)

The goal of sequential recommendation (SR) is to predict a user's potential interested items based on her/his historical interaction sequences. Most existing sequential recommenders are developed based on ID features, which, despite their widespread use, often underperform with sparse IDs and struggle with the cold-start problem. Besides, inconsistent ID mappings hinder the model's transferability, isolating similar recommendation domains that could have been co-optimized. This paper aims to address these issues by exploring the potential of multi-modal information in learning robust and generalizable sequence representations. We propose MISSRec, a multi-modal pre-training and transfer learning framework for SR. On the user side, we design a Transformer-based encoder-decoder model, where the contextual encoder learns to capture the sequence-level multi-modal user interests while a novel interest-aware decoder is developed to grasp item-modality-interest relations for better sequence representation. On the candidate item side, we adopt a dynamic fusion module to produce user-adaptive item representation, providing more precise matching between users and items. We pre-train the model with contrastive learning objectives and fine-tune it in an efficient manner. Extensive experiments demonstrate the effectiveness and flexibility of MISSRec, promising a practical solution for real-world recommendation scenarios. Data and code are available on urlhttps://github.com/gimpong/MM23-MISSRec.