# Exploring Recommendation Capabilities of GPT-4V(ision): A Preliminary Case Study arXiv: https://arxiv.org/abs/2311.04199 ยท HTML: https://mlsanity.com/paper/2311.04199/ ## Key facts - Cited by 12 papers in the map (data through 14 Oct 2026). - Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. - Builds on 22 of its 37 references that are in the map. - First posted on arXiv 7 Nov 2023. ## Placed in - Multimodal LLMs / vision-language models / [Document understanding and multimodal RAG](https://mlsanity.com/mllm/document-understanding-and-multimodal-rag/) ## Builds on (most cited first, 22 of 22) - [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023) - [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023) - [HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models](https://mlsanity.com/paper/2310.14566/) (2023) - [ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPT](https://mlsanity.com/paper/2302.10205/) (2023) - [Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender System](https://mlsanity.com/paper/2303.14524/) (2023) - [A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT](https://mlsanity.com/paper/2303.04226/) (2023) - [Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendation](https://mlsanity.com/paper/2305.07609/) (2023) - [Is ChatGPT a Good Sentiment Analyzer? A Preliminary Study](https://mlsanity.com/paper/2304.04339/) (2023) - [MM-VID: Advancing Video Understanding with GPT-4V(ision)](https://mlsanity.com/paper/2310.19773/) (2023) - [Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosis](https://mlsanity.com/paper/2310.09909/) (2023) - [Qilin-Med-VL: Towards Chinese Large Vision-Language Model for General Healthcare](https://mlsanity.com/paper/2310.17956/) (2023) - [Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remark](https://mlsanity.com/paper/2304.11085/) (2023) - [Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluation](https://mlsanity.com/paper/2310.16809/) (2023) - [Multimodal Recommender Systems: A Survey](https://mlsanity.com/paper/2302.03883/) (2023) - [ChatGraph: Interpretable Text Classification by Converting ChatGPT Knowledge to Graphs](https://mlsanity.com/paper/2305.03513/) (2023) - [Applying BERT and ChatGPT for Sentiment Analysis of Lyme Disease in Scientific Literature](https://mlsanity.com/paper/2302.06474/) (2023) - [Qilin-Med: Multi-stage Knowledge Injection Advanced Medical Large Language Model](https://mlsanity.com/paper/2310.09089/) (2023) - [Evaluating ChatGPT as a Recommender System: A Rigorous Approach](https://mlsanity.com/paper/2309.03613/) (2023) - [A Preliminary Study of ChatGPT on News Recommendation: Personalization, Provider Fairness, Fake News](https://mlsanity.com/paper/2306.10702/) (2023) - [A Comprehensive Survey on Multimodal Recommender Systems: Taxonomy, Evaluation, and Future Directions](https://mlsanity.com/paper/2302.04473/) (2023) - [Breaking the Bank with ChatGPT: Few-Shot Text Classification for Finance](https://mlsanity.com/paper/2308.14634/) (2023) - [Improving Large Language Models for Clinical Named Entity Recognition via Prompt Engineering](https://mlsanity.com/paper/2303.16416/) (2023) ## Built on by (most cited first, 12 of 12) - [FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts](https://mlsanity.com/paper/2311.05608/) (2023) - [How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Prompts](https://mlsanity.com/paper/2402.13220/) (2024) - [GPT-4V with Emotion: A Zero-shot Benchmark for Generalized Emotion Recognition](https://mlsanity.com/paper/2312.04293/) (2023) - [GPT4Vis: What Can GPT-4 Do for Zero-shot Visual Recognition?](https://mlsanity.com/paper/2311.15732/) (2023) - [Personalized Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2412.02142/) (2024) - [Exploring Boundary of GPT-4V on Marine Analysis: A Preliminary Case Study](https://mlsanity.com/paper/2401.02147/) (2024) - [GPT4Ego: Unleashing the Potential of Pre-trained Models for Zero-Shot Egocentric Action Recognition](https://mlsanity.com/paper/2401.10039/) (2024) - [GPT-4V(ision) as A Social Media Analysis Engine](https://mlsanity.com/paper/2311.07547/) (2023) - [Thinking with Drafts: Speculative Temporal Reasoning for Efficient Long Video Understanding](https://mlsanity.com/paper/2512.00805/) (2025) - [MarineEval: Assessing the Marine Intelligence of Vision-Language Models](https://mlsanity.com/paper/2512.21126/) (2025) - [Trustworthy Large Models in Vision: A Survey](https://mlsanity.com/paper/2311.09680/) (2023) - [Stateful Cooperative Agents Safeguarding LLMs Against Evolving Multi-Turn Attacks](https://mlsanity.com/paper/2608.00134/) (2026) ## Abstract (arXiv) Large Multimodal Models (LMMs) have demonstrated impressive performance across various vision and language tasks, yet their potential applications in recommendation tasks with visual assistance remain unexplored. To bridge this gap, we present a preliminary case study investigating the recommendation capabilities of GPT-4V(ison), a recently released LMM by OpenAI. We construct a series of qualitative test samples spanning multiple domains and employ these samples to assess the quality of GPT-4V's responses within recommendation scenarios. Evaluation results on these test samples prove that GPT-4V has remarkable zero-shot recommendation abilities across diverse domains, thanks to its robust visual-text comprehension capabilities and extensive general knowledge. However, we have also identified some limitations in using GPT-4V for recommendations, including a tendency to provide similar responses when given similar inputs. This report concludes with an in-depth discussion of the challenges and research opportunities associated with utilizing GPT-4V in recommendation scenarios. Our objective is to explore the potential of extending LMMs from vision and language tasks to recommendation tasks. We hope to inspire further research into next-generation multimodal generative recommendation models, which can enhance user experiences by offering greater diversity and interactivity. All images and prompts used in this report will be accessible at https://github.com/PALIN2018/Evaluate_GPT-4V_Rec.