mlsanityData to 7 Oct 2026

Rec-GPT4V: Multimodal Recommendation with Large Vision-Language Models

arXiv 2402.08670 ↗First posted 13 Feb 2024cs.AIcited by 18

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

The development of large vision-language models (LVLMs) offers the potential to address challenges faced by traditional multimodal recommendations thanks to their proficient understanding of static images and textual dynamics. However, the application of LVLMs in this field is still limited due to the following complexities: First, LVLMs lack user preference knowledge as they are trained from vast general datasets. Second, LVLMs suffer setbacks in addressing multiple image dynamics in scenarios involving discrete, noisy, and redundant image sequences. To overcome these issues, we propose the novel reasoning scheme named Rec-GPT4V: Visual-Summary Thought (VST) of leveraging large vision-language models for multimodal recommendation. We utilize user history as in-context user preferences to address the first challenge. Next, we prompt LVLMs to generate item image summaries and utilize image comprehension in natural language space combined with item titles to query the user preferences over candidate items. We conduct comprehensive experiments across four datasets with three LVLMs: GPT4-V, LLaVa-7b, and LLaVa-13b. The numerical results indicate the efficacy of VST.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 18 of its 39 references that are in the map. First posted on arXiv 13 Feb 2024.

Builds on

18 of 39 references.

References · 18

  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  2. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  3. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  4. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  5. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  6. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  7. 2023Large Language Models are Zero-Shot Rankers for Recommender Systemscited by 307
  8. 2015VBPR: Visual Bayesian Personalized Ranking from Implicit Feedbackcited by 151
  9. 2023On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Drivingcited by 79
  10. 2022Bootstrap Latent Representations for Multi-modal Recommendationcited by 55
  11. 2022A Tale of Two Graphs: Freezing and Denoising Graph Structures for Multimodal Recommendationcited by 53
  12. 2023VIP5: Towards Multimodal Foundation Models for Recommendationcited by 45
6 more
  1. 2023Towards Generic Anomaly Detection and Understanding: Large-scale Visual-linguistic Model (GPT-4V) Takes the Leadcited by 40
  2. 2023GPT4Video: A Unified Multimodal Large Language Model for lnstruction-Followed Understanding and Safety-Aware Generationcited by 36
  3. 2023DRDT: Dynamic Reflection with Divergent Thinking for LLM-based Sequential Recommendationcited by 27
  4. 2024ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Modelscited by 19
  5. 2021Latent Structure Mining with Contrastive Modality Fusion for Multimedia Recommendationcited by 18
  6. 2023Attention-guided Multi-step Fusion: A Hierarchical Fusion Network for Multimodal Recommendationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 11; 2026: 5.

18 papers.

Sort:

By area:

  1. 2025SimUSER: Simulating User Behavior with Large Language Models for Recommender System Evaluationcited by 32
  2. 2024Personalized Multimodal Large Language Models: A Surveycited by 19
  3. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  4. 2025Structured Spectral Reasoning for Frequency-Adaptive Multimodal Recommendationcited by 6
  5. 2024Less is More: A Simple yet Effective Token Reduction Method for Efficient Multi-modal LLMscited by 3
  6. 2025RAG-VisualRec: An Open Resource for Vision- and Text-Enhanced Retrieval-Augmented Generation in Recommendationcited by 3
  7. 2026Multimodal Generative Engine Optimization: Rank Manipulation for Vision-Language Model Rankerscited by 3
  8. 2025Cache-of-Thought: Master-Apprentice Framework for Cost-Effective Vision Language Model Reasoningcited by 2
  9. 2025REGen: Multimodal Retrieval-Embedded Generation for Long-to-Short Video Editingcited by 1
  10. 2025Serendipitous Recommendation with Multimodal LLMcited by 1
  11. 2025Timeripple: Accelerating vDiTs by Understanding the Spatio-Temporal Correlations in Latent Spacecited by 1
  12. 2025LLM Reasoning for Cold-Start Item Recommendationcited by 1
6 more
  1. 2026Agent4POI: Agentic Context-Conditioned Affordance Reasoning for Multimodal Point-of-Interest Recommendationcited by 1
  2. 2025GTS-LUM: Reshaping User Behavior Modeling with LLMs in Telecommunications Industrycited by 0
  3. 2025Improving LLM Interpretability and Performance via Guided Embedding Refinement for Sequential Recommendationcited by 0
  4. 2026MLLMRec-R1: Incentivizing Reasoning Capability in Large Language Models for Multimodal Sequential Recommendationcited by 0
  5. 2026Seeing Red, Thinking Bad: Color Bias in Vision Language Modelscited by 0
  6. 2026Multimodal Graph Retrieval-Augmented Sequential Recommendation via Collaborative Filtering Pathscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026VLM4Rec: Multimodal Semantic Representation for Recommendation with Large Vision-Language Models
  2. 2023Exploring Recommendation Capabilities of GPT-4V(ision): A Preliminary Case Study
  3. 2024Imagery as Inquiry: Exploring A Multimodal Dataset for Conversational Recommendation
  4. 2025Multimodal Large Language Models with Adaptive Preference Optimization for Sequential Recommendation
  5. 2026OpticalRec: Unified Optical Vision-Language Representation for Multimodal Recommendation
  6. 2024Harnessing Multimodal Large Language Models for Multimodal Sequential Recommendation
  7. 2024NoteLLM-2: Multimodal Large Representation Models for Recommendation
  8. 2025Do Recommender Systems Really Leverage Multimodal Content? A Comprehensive Analysis on Multimodal Representations for Recommendation