VLM4Rec: Multimodal Semantic Representation for Recommendation with Large Vision-Language Models
On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG
Abstract (arXiv)
Multimodal recommendation is commonly framed as a feature fusion problem, where textual and visual signals are combined to better model user preference. However, the effectiveness of multimodal recommendation may depend not only on how modalities are fused, but also on whether item content is represented in a semantic space aligned with preference matching. This issue is particularly important because raw visual features often preserve appearance similarity, while user decisions are typically driven by higher-level semantic factors such as style, material, and usage context. Motivated by this observation, we propose LVLM-grounded Multimodal Semantic Representation for Recommendation (VLM4Rec), a lightweight framework that organizes multimodal item content through semantic alignment rather than direct feature fusion. VLM4Rec first uses a large vision-language model to ground each item image into an explicit natural-language description, and then encodes the grounded semantics into dense item representations for preference-oriented retrieval. Recommendation is subsequently performed through a simple profile-based semantic matching mechanism over historical item embeddings, yielding a practical offline-online decomposition. Extensive experiments on multiple multimodal recommendation datasets show that VLM4Rec consistently improves performance over raw visual features and several fusion-based alternatives, suggesting that representation quality may matter more than fusion complexity in this setting. The code is released at https://github.com/tyvalencia/enhancing-mm-rec-sys.
Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 57 references that are in the map. First posted on arXiv 13 Mar 2026.
Builds on
29 of 57 references.
References · 29
- 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 515
- 2022Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)cited by 397
- 2023TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendationcited by 336
- 2021Are Graph Augmentations Necessary? Simple Graph Contrastive Learning for Recommendationcited by 173
- 2023LLMRec: Large Language Models with Graph Augmentation for Recommendationcited by 172
- 2015VBPR: Visual Bayesian Personalized Ranking from Implicit Feedbackcited by 151
- 2023Text Is All You Need: Learning Language Representations for Sequential Recommendationcited by 149
- 2023RecMind: Large Language Model Powered Agent For Recommendationcited by 129
- 2021Mining Latent Structures for Multimedia Recommendationcited by 79
- 2021GRCN: Graph-Refined Convolutional Network for Multimedia Recommendation with Implicit Feedbackcited by 58
- 2023LlamaRec: Two-Stage Recommendation using Large Language Models for Rankingcited by 53
- 2024TourRank: Utilizing Large Language Models for Documents Ranking with a Tournament-Inspired Strategycited by 47
17 more
- 2017Visually-Aware Fashion Recommendation and Design with Generative Image Modelscited by 43
- 2023Multi-View Graph Convolutional Network for Multimedia Recommendationcited by 32
- 2024TS-Reasoner: Domain-Oriented Time Series Inference Agents for Reasoning and Automated Analysiscited by 28
- 2024Rethinking Large Language Model Architectures for Sequential Recommendationscited by 27
- 2024CoST: Contrastive Quantization based Semantic Tokenization for Generative Recommendationcited by 26
- 2024Multimodal Pretraining, Adaptation, and Generation for Recommendation: A Surveycited by 24
- 2024DiffMM: Multi-Modal Diffusion Model for Recommendationcited by 21
- 2024AlignRec: Aligning and Training in Multimodal Recommendationscited by 20
- 2025HDLCoRe: A Training-Free Framework for Mitigating Hallucinations in LLM-Generated HDLcited by 17
- 2025R4ec: A Reasoning, Reflection, and Refinement Framework for Recommendation Systemscited by 16
- 2024PromptMM: Multi-Modal Knowledge Distillation for Recommendation with Prompt-Tuningcited by 14
- 2025Maestro: Learning to Collaborate via Conditional Listwise Policy Optimization for Multi-Agent LLMscited by 14
- 2024Spectrum-based Modality Representation Fusion Graph Convolutional Network for Multimodal Recommendationcited by 13
- 2026Self-Compression of Chain-of-Thought via Multi-Agent Reinforcement Learningcited by 13
- 2025Structured Spectral Reasoning for Frequency-Adaptive Multimodal Recommendationcited by 6
- 2026FITMM: Adaptive Frequency-Aware Multimodal Recommendation via Information-Theoretic Representation Learningcited by 6
- 2026RecGOAT: Graph Optimal Adaptive Transport for LLM-Enhanced Multimodal Recommendation with Dual Semantic Alignmentcited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Do Recommender Systems Really Leverage Multimodal Content? A Comprehensive Analysis on Multimodal Representations for Recommendation
- 2024Rec-GPT4V: Multimodal Recommendation with Large Vision-Language Models
- 2025MLLMRec: A Preference Reasoning Paradigm with Graph Refinement for Multimodal Recommendation
- 2026OpticalRec: Unified Optical Vision-Language Representation for Multimodal Recommendation
- 2025Structural and Disentangled Adaptation of Large Vision Language Models for Multimodal Recommendation
- 2024NoteLLM-2: Multimodal Large Representation Models for Recommendation
- 2026VLM2Rec: Resolving Modality Collapse in Vision-Language Model Embedders for Multimodal Sequential Recommendation
- 2025MMSRARec: Summarization and Retrieval Augumented Sequential Recommendation Based on Multimodal Large Language Model