# Serendipitous Recommendation with Multimodal LLM

arXiv: https://arxiv.org/abs/2506.08283 · HTML: https://mlsanity.com/paper/2506.08283/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 14 of its 30 references that are in the map.
- First posted on arXiv 9 Jun 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Document understanding and multimodal RAG](https://mlsanity.com/mllm/document-understanding-and-multimodal-rag/)

## Builds on (most cited first, 14 of 14)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation](https://mlsanity.com/paper/2305.00447/) (2023)
- [Is ChatGPT a Good Recommender? A Preliminary Study](https://mlsanity.com/paper/2304.10149/) (2023)
- [Uncovering ChatGPT's Capabilities in Recommender Systems](https://mlsanity.com/paper/2305.02182/) (2023)
- [GPT4Rec: A Generative Framework for Personalized Recommendation and User Interests Interpretation](https://mlsanity.com/paper/2304.03879/) (2023)
- [Towards Open-World Recommendation with Knowledge Augmentation from Large Language Models](https://mlsanity.com/paper/2306.10933/) (2023)
- [ONCE: Boosting Content-based Recommendation with Both Open- and Closed-source Large Language Models](https://mlsanity.com/paper/2305.06566/) (2023)
- [VIP5: Towards Multimodal Foundation Models for Recommendation](https://mlsanity.com/paper/2305.14302/) (2023)
- [NoteLLM-2: Multimodal Large Representation Models for Recommendation](https://mlsanity.com/paper/2405.16789/) (2024)
- [Large Language Models as Data Augmenters for Cold-Start Item Recommendation](https://mlsanity.com/paper/2402.11724/) (2024)
- [Rec-GPT4V: Multimodal Recommendation with Large Vision-Language Models](https://mlsanity.com/paper/2402.08670/) (2024)
- [TagGPT: Large Language Models are Zero-shot Multimodal Taggers](https://mlsanity.com/paper/2304.03022/) (2023)

## Built on by (most cited first, 1 of 1)

- [When Relevance Meets Novelty: Dual-Stable Periodic Optimization for Serendipitous Recommendation](https://mlsanity.com/paper/2508.00450/) (2025)

## Abstract (arXiv)

Conventional recommendation systems succeed in identifying relevant content but often fail to provide users with surprising or novel items. Multimodal Large Language Models (MLLMs) possess the world knowledge and multimodal understanding needed for serendipity, but their integration into billion-item-scale platforms presents significant challenges. In this paper, we propose a novel hierarchical framework where fine-tuned MLLMs provide high-level guidance to conventional recommendation models, steering them towards more serendipitous suggestions. This approach leverages MLLM strengths in understanding multimodal content and user interests while retaining the efficiency of traditional models for item-level recommendation. This mitigates the complexity of applying MLLMs directly to vast action spaces. We also demonstrate a chain-of-thought strategy enabling MLLMs to discover novel user interests by first understanding video content and then identifying relevant yet unexplored interest clusters. Through live experiments within a commercial short-form video platform serving billions of users, we show that our MLLM-powered approach significantly improves both recommendation serendipity and user satisfaction.