# VLM4Rec: Multimodal Semantic Representation for Recommendation with Large Vision-Language Models

arXiv: https://arxiv.org/abs/2603.12625 · HTML: https://mlsanity.com/paper/2603.12625/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 29 of its 57 references that are in the map.
- First posted on arXiv 13 Mar 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Document understanding and multimodal RAG](https://mlsanity.com/mllm/document-understanding-and-multimodal-rag/)

## Builds on (most cited first, 29 of 29)

- [MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action](https://mlsanity.com/paper/2303.11381/) (2023)
- [Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)](https://mlsanity.com/paper/2203.13366/) (2022)
- [TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation](https://mlsanity.com/paper/2305.00447/) (2023)
- [Are Graph Augmentations Necessary? Simple Graph Contrastive Learning for Recommendation](https://mlsanity.com/paper/2112.08679/) (2021)
- [LLMRec: Large Language Models with Graph Augmentation for Recommendation](https://mlsanity.com/paper/2311.00423/) (2023)
- [VBPR: Visual Bayesian Personalized Ranking from Implicit Feedback](https://mlsanity.com/paper/1510.01784/) (2015)
- [Text Is All You Need: Learning Language Representations for Sequential Recommendation](https://mlsanity.com/paper/2305.13731/) (2023)
- [RecMind: Large Language Model Powered Agent For Recommendation](https://mlsanity.com/paper/2308.14296/) (2023)
- [Mining Latent Structures for Multimedia Recommendation](https://mlsanity.com/paper/2104.09036/) (2021)
- [GRCN: Graph-Refined Convolutional Network for Multimedia Recommendation with Implicit Feedback](https://mlsanity.com/paper/2111.02036/) (2021)
- [LlamaRec: Two-Stage Recommendation using Large Language Models for Ranking](https://mlsanity.com/paper/2311.02089/) (2023)
- [TourRank: Utilizing Large Language Models for Documents Ranking with a Tournament-Inspired Strategy](https://mlsanity.com/paper/2406.11678/) (2024)
- [Visually-Aware Fashion Recommendation and Design with Generative Image Models](https://mlsanity.com/paper/1711.02231/) (2017)
- [Multi-View Graph Convolutional Network for Multimedia Recommendation](https://mlsanity.com/paper/2308.03588/) (2023)
- [TS-Reasoner: Domain-Oriented Time Series Inference Agents for Reasoning and Automated Analysis](https://mlsanity.com/paper/2410.04047/) (2024)
- [Rethinking Large Language Model Architectures for Sequential Recommendations](https://mlsanity.com/paper/2402.09543/) (2024)
- [CoST: Contrastive Quantization based Semantic Tokenization for Generative Recommendation](https://mlsanity.com/paper/2404.14774/) (2024)
- [Multimodal Pretraining, Adaptation, and Generation for Recommendation: A Survey](https://mlsanity.com/paper/2404.00621/) (2024)
- [DiffMM: Multi-Modal Diffusion Model for Recommendation](https://mlsanity.com/paper/2406.11781/) (2024)
- [AlignRec: Aligning and Training in Multimodal Recommendations](https://mlsanity.com/paper/2403.12384/) (2024)
- [HDLCoRe: A Training-Free Framework for Mitigating Hallucinations in LLM-Generated HDL](https://mlsanity.com/paper/2503.16528/) (2025)
- [R4ec: A Reasoning, Reflection, and Refinement Framework for Recommendation Systems](https://mlsanity.com/paper/2507.17249/) (2025)
- [PromptMM: Multi-Modal Knowledge Distillation for Recommendation with Prompt-Tuning](https://mlsanity.com/paper/2402.17188/) (2024)
- [Maestro: Learning to Collaborate via Conditional Listwise Policy Optimization for Multi-Agent LLMs](https://mlsanity.com/paper/2511.06134/) (2025)
- [Spectrum-based Modality Representation Fusion Graph Convolutional Network for Multimodal Recommendation](https://mlsanity.com/paper/2412.14978/) (2024)
- [Self-Compression of Chain-of-Thought via Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2601.21919/) (2026)
- [Structured Spectral Reasoning for Frequency-Adaptive Multimodal Recommendation](https://mlsanity.com/paper/2512.01372/) (2025)
- [FITMM: Adaptive Frequency-Aware Multimodal Recommendation via Information-Theoretic Representation Learning](https://mlsanity.com/paper/2601.22498/) (2026)
- [RecGOAT: Graph Optimal Adaptive Transport for LLM-Enhanced Multimodal Recommendation with Dual Semantic Alignment](https://mlsanity.com/paper/2602.00682/) (2026)

## Abstract (arXiv)

Multimodal recommendation is commonly framed as a feature fusion problem, where textual and visual signals are combined to better model user preference. However, the effectiveness of multimodal recommendation may depend not only on how modalities are fused, but also on whether item content is represented in a semantic space aligned with preference matching. This issue is particularly important because raw visual features often preserve appearance similarity, while user decisions are typically driven by higher-level semantic factors such as style, material, and usage context. Motivated by this observation, we propose LVLM-grounded Multimodal Semantic Representation for Recommendation (VLM4Rec), a lightweight framework that organizes multimodal item content through semantic alignment rather than direct feature fusion. VLM4Rec first uses a large vision-language model to ground each item image into an explicit natural-language description, and then encodes the grounded semantics into dense item representations for preference-oriented retrieval. Recommendation is subsequently performed through a simple profile-based semantic matching mechanism over historical item embeddings, yielding a practical offline-online decomposition. Extensive experiments on multiple multimodal recommendation datasets show that VLM4Rec consistently improves performance over raw visual features and several fusion-based alternatives, suggesting that representation quality may matter more than fusion complexity in this setting. The code is released at https://github.com/tyvalencia/enhancing-mm-rec-sys.