mlsanityData to 14 Oct 2026

GPT4Vis: What Can GPT-4 Do for Zero-shot Visual Recognition?

arXiv 2311.15732 ↗First posted 27 Nov 2023cs.CVcited by 35

Not on the map. Why?

Cited by 22 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper does not present a novel method. Instead, it delves into an essential, yet must-know baseline in light of the latest advancements in Generative Artificial Intelligence (GenAI): the utilization of GPT-4 for visual understanding. Our study centers on the evaluation of GPT-4's linguistic and visual capabilities in zero-shot visual recognition tasks: Firstly, we explore the potential of its generated rich textual descriptions across various categories to enhance recognition performance without any training. Secondly, we evaluate GPT-4's visual proficiency in directly recognizing diverse visual content. We conducted extensive experiments to systematically evaluate GPT-4's performance across images, videos, and point clouds, using 16 benchmark datasets to measure top-1 and top-5 accuracy. Our findings show that GPT-4, enhanced with rich linguistic descriptions, significantly improves zero-shot recognition, offering an average top-1 accuracy increase of 7% across all datasets. GPT-4 excels in visual recognition, outshining OpenAI-CLIP's ViT-L and rivaling EVA-CLIP's ViT-E, particularly in video datasets HMDB-51 and UCF-101, where it leads by 22% and 9%, respectively. We hope this research contributes valuable data points and experience for future studies. We release our code at https://github.com/whwu95/GPT4Vis.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 28 of its 44 references that are in the map. First posted on arXiv 27 Nov 2023.

Builds on

28 of 44 references.

References · 28

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  5. 2013Describing Textures in the Wildcited by 2,125
  6. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  7. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  8. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  9. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  10. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  11. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  12. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 651
16 more
  1. 2015Multi-view Convolutional Neural Networks for 3D Shape Recognitioncited by 565
  2. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 463
  3. 2022Visual Classification via Description from Large Language Modelscited by 336
  4. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 320
  5. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 187
  6. 2022Cap4Video: What Can Auxiliary Captions Do for Text-Video Retrieval?cited by 90
  7. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 83
  8. 2023On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Drivingcited by 80
  9. 2022Revisiting Classifier: Transferring Vision-Language Models for Video Recognitioncited by 71
  10. 2023UATVR: Uncertainty-Adaptive Text-Video Retrievalcited by 55
  11. 2022Bidirectional Cross-Modal Knowledge Exploration for Video Recognition with Pre-trained Vision-Language Modelscited by 49
  12. 2023Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluationcited by 46
  13. 2023Towards Generic Anomaly Detection and Understanding: Large-scale Visual-linguistic Model (GPT-4V) Takes the Leadcited by 40
  14. 2023A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysiscited by 37
  15. 2023ChatGPT-Powered Hierarchical Comparisons for Image Classificationcited by 22
  16. 2023Exploring Recommendation Capabilities of GPT-4V(ision): A Preliminary Case Studycited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 21; 2025: 7; 2026: 6.

35 papers.

Sort:

By area:

  1. 2024Why are Visually-Grounded Language Models Bad at Image Classification?cited by 102
  2. 2024Dense Connector for MLLMscited by 49
  3. 2023GPT-4V with Emotion: A Zero-shot Benchmark for Generalized Emotion Recognitioncited by 38
  4. 2024Agent3D-Zero: An Agent for Zero-shot 3D Understandingcited by 29
  5. 2024FreeVA: Offline MLLM as Training-Free Video Assistantcited by 29
  6. 2024Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correctingcited by 26
  7. 2024RAR: Retrieving And Ranking Augmented MLLMs for Visual Recognitioncited by 24
  8. 2025A Survey on Agentic Multimodal Large Language Modelscited by 23
  9. 2024Holistic Unlearning Benchmark: A Multi-Faceted Evaluation for Text-to-Image Diffusion Model Unlearningcited by 21
  10. 2024Enhancing Zero-Shot Facial Expression Recognition by LLM Knowledge Transfercited by 18
  11. 2024Can Textual Semantics Mitigate Sounding Object Segmentation Preference?cited by 13
  12. 2025ProAPO: Progressively Automatic Prompt Optimization for Visual Classificationcited by 13
23 more
  1. 2024Bootstrapping Cognitive Agents with a Large Language Modelcited by 9
  2. 2024GPT4Ego: Unleashing the Potential of Pre-trained Models for Zero-Shot Egocentric Action Recognitioncited by 8
  3. 2024Enhancing Vision-Language Few-Shot Adaptation with Negative Learningcited by 8
  4. 2024MFE-ETP: A Comprehensive Evaluation Benchmark for Multi-modal Foundation Models on Embodied Task Planningcited by 8
  5. 2024TG-LLaVA: Text Guided LLaVA via Learnable Latent Embeddingscited by 8
  6. 2025Visual Language Models show widespread visual deficits on neuropsychological testscited by 3
  7. 2024CollagePrompt: A Benchmark for Budget-Friendly Visual Recognition with GPT-4Vcited by 2
  8. 2024Can Large Language Models Grasp Event Signals? Exploring Pure Zero-Shot Event-based Recognitioncited by 2
  9. 2024Do large language vision models understand 3D shapes?cited by 2
  10. 2025MADS: Multi-Attribute Document Supervision for Zero-Shot Image Classificationcited by 2
  11. 2026Multimodal Large Language Models as Image Classifierscited by 2
  12. 2026PReD: An LLM-based Foundation Multimodal Model for Electromagnetic Perception, Recognition, and Decisioncited by 2
  13. 2024GPT-4V Explorations: Mining Autonomous Drivingcited by 1
  14. 2024Blocks as Probes: Dissecting Categorization Ability of Large Multimodal Modelscited by 1
  15. 2024Explainable Search and Discovery of Visual Cultural Heritage Collections with Multimodal Large Language Modelscited by 1
  16. 2025Enhancing Spatio-Temporal Zero-shot Action Recognition with Language-driven Description Attributescited by 1
  17. 2026GhostUI: Unveiling Hidden Interactions in Mobile UIcited by 1
  18. 2024Constructing Multilingual Visual-Text Datasets Revealing Visual Multilingual Ability of Vision Language Modelscited by 0
  19. 2025Large Language Models Facilitate Vision Reflection in Image Classificationcited by 0
  20. 2025Auxiliary Descriptive Knowledge for Few-Shot Adaptation of Vision-Language Modelcited by 0
  21. 2026AdaBoosting Text Prompts for Vision-Language Modelscited by 0
  22. 2026Can Vision-Language Models Judge Olympic Diving? From Reasoning to Scores in Zero-Shot Action Quality Assessmentcited by 0
  23. 2026Revisiting Natural Language Prompts from Large Language Models for Open-Vocabulary Video Recognitioncited by 0