# Rec-GPT4V: Multimodal Recommendation with Large Vision-Language Models

arXiv: https://arxiv.org/abs/2402.08670 · HTML: https://mlsanity.com/paper/2402.08670/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 18 of its 39 references that are in the map.
- First posted on arXiv 13 Feb 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Document understanding and multimodal RAG](https://mlsanity.com/mllm/document-understanding-and-multimodal-rag/)

## Builds on (most cited first, 18 of 18)

- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions](https://mlsanity.com/paper/2212.10509/) (2022)
- [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021)
- [Large Language Models are Zero-Shot Rankers for Recommender Systems](https://mlsanity.com/paper/2305.08845/) (2023)
- [VBPR: Visual Bayesian Personalized Ranking from Implicit Feedback](https://mlsanity.com/paper/1510.01784/) (2015)
- [On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Driving](https://mlsanity.com/paper/2311.05332/) (2023)
- [Bootstrap Latent Representations for Multi-modal Recommendation](https://mlsanity.com/paper/2207.05969/) (2022)
- [A Tale of Two Graphs: Freezing and Denoising Graph Structures for Multimodal Recommendation](https://mlsanity.com/paper/2211.06924/) (2022)
- [VIP5: Towards Multimodal Foundation Models for Recommendation](https://mlsanity.com/paper/2305.14302/) (2023)
- [Towards Generic Anomaly Detection and Understanding: Large-scale Visual-linguistic Model (GPT-4V) Takes the Lead](https://mlsanity.com/paper/2311.02782/) (2023)
- [GPT4Video: A Unified Multimodal Large Language Model for lnstruction-Followed Understanding and Safety-Aware Generation](https://mlsanity.com/paper/2311.16511/) (2023)
- [DRDT: Dynamic Reflection with Divergent Thinking for LLM-based Sequential Recommendation](https://mlsanity.com/paper/2312.11336/) (2023)
- [ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Models](https://mlsanity.com/paper/2401.13311/) (2024)
- [Latent Structure Mining with Contrastive Modality Fusion for Multimedia Recommendation](https://mlsanity.com/paper/2111.00678/) (2021)
- [Attention-guided Multi-step Fusion: A Hierarchical Fusion Network for Multimodal Recommendation](https://mlsanity.com/paper/2304.11979/) (2023)

## Built on by (most cited first, 18 of 18)

- [SimUSER: Simulating User Behavior with Large Language Models for Recommender System Evaluation](https://mlsanity.com/paper/2504.12722/) (2025)
- [A Survey on Agentic Multimodal Large Language Models](https://mlsanity.com/paper/2510.10991/) (2025)
- [Personalized Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2412.02142/) (2024)
- [Structured Spectral Reasoning for Frequency-Adaptive Multimodal Recommendation](https://mlsanity.com/paper/2512.01372/) (2025)
- [Less is More: A Simple yet Effective Token Reduction Method for Efficient Multi-modal LLMs](https://mlsanity.com/paper/2409.10994/) (2024)
- [Cache-of-Thought: Master-Apprentice Framework for Cost-Effective Vision Language Model Reasoning](https://mlsanity.com/paper/2502.20587/) (2025)
- [RAG-VisualRec: An Open Resource for Vision- and Text-Enhanced Retrieval-Augmented Generation in Recommendation](https://mlsanity.com/paper/2506.20817/) (2025)
- [Multimodal Generative Engine Optimization: Rank Manipulation for Vision-Language Model Rankers](https://mlsanity.com/paper/2601.12263/) (2026)
- [REGen: Multimodal Retrieval-Embedded Generation for Long-to-Short Video Editing](https://mlsanity.com/paper/2505.18880/) (2025)
- [Serendipitous Recommendation with Multimodal LLM](https://mlsanity.com/paper/2506.08283/) (2025)
- [Timeripple: Accelerating vDiTs by Understanding the Spatio-Temporal Correlations in Latent Space](https://mlsanity.com/paper/2511.12035/) (2025)
- [LLM Reasoning for Cold-Start Item Recommendation](https://mlsanity.com/paper/2511.18261/) (2025)
- [Agent4POI: Agentic Context-Conditioned Affordance Reasoning for Multimodal Point-of-Interest Recommendation](https://mlsanity.com/paper/2605.15203/) (2026)
- [GTS-LUM: Reshaping User Behavior Modeling with LLMs in Telecommunications Industry](https://mlsanity.com/paper/2504.06511/) (2025)
- [Improving LLM Interpretability and Performance via Guided Embedding Refinement for Sequential Recommendation](https://mlsanity.com/paper/2504.11658/) (2025)
- [MLLMRec-R1: Incentivizing Reasoning Capability in Large Language Models for Multimodal Sequential Recommendation](https://mlsanity.com/paper/2603.06243/) (2026)
- [Seeing Red, Thinking Bad: Color Bias in Vision Language Models](https://mlsanity.com/paper/2608.14286/) (2026)
- [Multimodal Graph Retrieval-Augmented Sequential Recommendation via Collaborative Filtering Paths](https://mlsanity.com/paper/2610.11228/) (2026)

## Abstract (arXiv)

The development of large vision-language models (LVLMs) offers the potential to address challenges faced by traditional multimodal recommendations thanks to their proficient understanding of static images and textual dynamics. However, the application of LVLMs in this field is still limited due to the following complexities: First, LVLMs lack user preference knowledge as they are trained from vast general datasets. Second, LVLMs suffer setbacks in addressing multiple image dynamics in scenarios involving discrete, noisy, and redundant image sequences. To overcome these issues, we propose the novel reasoning scheme named Rec-GPT4V: Visual-Summary Thought (VST) of leveraging large vision-language models for multimodal recommendation. We utilize user history as in-context user preferences to address the first challenge. Next, we prompt LVLMs to generate item image summaries and utilize image comprehension in natural language space combined with item titles to query the user preferences over candidate items. We conduct comprehensive experiments across four datasets with three LVLMs: GPT4-V, LLaVa-7b, and LLaVa-13b. The numerical results indicate the efficacy of VST.