# GPT4Vis: What Can GPT-4 Do for Zero-shot Visual Recognition?

arXiv: https://arxiv.org/abs/2311.15732 · HTML: https://mlsanity.com/paper/2311.15732/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 44 references that are in the map.
- First posted on arXiv 27 Nov 2023.

## Builds on (most cited first, 28 of 28)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [EVA-CLIP: Improved Training Techniques for CLIP at Scale](https://mlsanity.com/paper/2303.15389/) (2023)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [Multi-view Convolutional Neural Networks for 3D Shape Recognition](https://mlsanity.com/paper/1505.00880/) (2015)
- [PointCLIP: Point Cloud Understanding by CLIP](https://mlsanity.com/paper/2112.02413/) (2021)
- [Visual Classification via Description from Large Language Models](https://mlsanity.com/paper/2210.07183/) (2022)
- [ActionCLIP: A New Paradigm for Video Action Recognition](https://mlsanity.com/paper/2109.08472/) (2021)
- [PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learning](https://mlsanity.com/paper/2211.11682/) (2022)
- [Cap4Video: What Can Auxiliary Captions Do for Text-Video Retrieval?](https://mlsanity.com/paper/2301.00184/) (2022)
- [MM-VID: Advancing Video Understanding with GPT-4V(ision)](https://mlsanity.com/paper/2310.19773/) (2023)
- [On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Driving](https://mlsanity.com/paper/2311.05332/) (2023)
- [Revisiting Classifier: Transferring Vision-Language Models for Video Recognition](https://mlsanity.com/paper/2207.01297/) (2022)
- [UATVR: Uncertainty-Adaptive Text-Video Retrieval](https://mlsanity.com/paper/2301.06309/) (2023)
- [Bidirectional Cross-Modal Knowledge Exploration for Video Recognition with Pre-trained Vision-Language Models](https://mlsanity.com/paper/2301.00182/) (2022)
- [Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluation](https://mlsanity.com/paper/2310.16809/) (2023)
- [Towards Generic Anomaly Detection and Understanding: Large-scale Visual-linguistic Model (GPT-4V) Takes the Lead](https://mlsanity.com/paper/2311.02782/) (2023)
- [A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysis](https://mlsanity.com/paper/2310.20381/) (2023)
- [ChatGPT-Powered Hierarchical Comparisons for Image Classification](https://mlsanity.com/paper/2311.00206/) (2023)
- [Exploring Recommendation Capabilities of GPT-4V(ision): A Preliminary Case Study](https://mlsanity.com/paper/2311.04199/) (2023)

## Built on by (most cited first, 30 of 35)

- [Why are Visually-Grounded Language Models Bad at Image Classification?](https://mlsanity.com/paper/2405.18415/) (2024)
- [Dense Connector for MLLMs](https://mlsanity.com/paper/2405.13800/) (2024)
- [GPT-4V with Emotion: A Zero-shot Benchmark for Generalized Emotion Recognition](https://mlsanity.com/paper/2312.04293/) (2023)
- [Agent3D-Zero: An Agent for Zero-shot 3D Understanding](https://mlsanity.com/paper/2403.11835/) (2024)
- [FreeVA: Offline MLLM as Training-Free Video Assistant](https://mlsanity.com/paper/2405.07798/) (2024)
- [Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correcting](https://mlsanity.com/paper/2410.19294/) (2024)
- [RAR: Retrieving And Ranking Augmented MLLMs for Visual Recognition](https://mlsanity.com/paper/2403.13805/) (2024)
- [A Survey on Agentic Multimodal Large Language Models](https://mlsanity.com/paper/2510.10991/) (2025)
- [Holistic Unlearning Benchmark: A Multi-Faceted Evaluation for Text-to-Image Diffusion Model Unlearning](https://mlsanity.com/paper/2410.05664/) (2024)
- [Enhancing Zero-Shot Facial Expression Recognition by LLM Knowledge Transfer](https://mlsanity.com/paper/2405.19100/) (2024)
- [Can Textual Semantics Mitigate Sounding Object Segmentation Preference?](https://mlsanity.com/paper/2407.10947/) (2024)
- [ProAPO: Progressively Automatic Prompt Optimization for Visual Classification](https://mlsanity.com/paper/2502.19844/) (2025)
- [Bootstrapping Cognitive Agents with a Large Language Model](https://mlsanity.com/paper/2403.00810/) (2024)
- [GPT4Ego: Unleashing the Potential of Pre-trained Models for Zero-Shot Egocentric Action Recognition](https://mlsanity.com/paper/2401.10039/) (2024)
- [Enhancing Vision-Language Few-Shot Adaptation with Negative Learning](https://mlsanity.com/paper/2403.12964/) (2024)
- [MFE-ETP: A Comprehensive Evaluation Benchmark for Multi-modal Foundation Models on Embodied Task Planning](https://mlsanity.com/paper/2407.05047/) (2024)
- [TG-LLaVA: Text Guided LLaVA via Learnable Latent Embeddings](https://mlsanity.com/paper/2409.09564/) (2024)
- [Visual Language Models show widespread visual deficits on neuropsychological tests](https://mlsanity.com/paper/2504.10786/) (2025)
- [CollagePrompt: A Benchmark for Budget-Friendly Visual Recognition with GPT-4V](https://mlsanity.com/paper/2403.11468/) (2024)
- [Can Large Language Models Grasp Event Signals? Exploring Pure Zero-Shot Event-based Recognition](https://mlsanity.com/paper/2409.09628/) (2024)
- [Do large language vision models understand 3D shapes?](https://mlsanity.com/paper/2412.10908/) (2024)
- [MADS: Multi-Attribute Document Supervision for Zero-Shot Image Classification](https://mlsanity.com/paper/2503.06847/) (2025)
- [Multimodal Large Language Models as Image Classifiers](https://mlsanity.com/paper/2603.06578/) (2026)
- [PReD: An LLM-based Foundation Multimodal Model for Electromagnetic Perception, Recognition, and Decision](https://mlsanity.com/paper/2603.28183/) (2026)
- [GPT-4V Explorations: Mining Autonomous Driving](https://mlsanity.com/paper/2406.16817/) (2024)
- [Blocks as Probes: Dissecting Categorization Ability of Large Multimodal Models](https://mlsanity.com/paper/2409.01560/) (2024)
- [Explainable Search and Discovery of Visual Cultural Heritage Collections with Multimodal Large Language Models](https://mlsanity.com/paper/2411.04663/) (2024)
- [Enhancing Spatio-Temporal Zero-shot Action Recognition with Language-driven Description Attributes](https://mlsanity.com/paper/2510.27255/) (2025)
- [GhostUI: Unveiling Hidden Interactions in Mobile UI](https://mlsanity.com/paper/2601.19258/) (2026)
- [Constructing Multilingual Visual-Text Datasets Revealing Visual Multilingual Ability of Vision Language Models](https://mlsanity.com/paper/2406.15359/) (2024)

## Abstract (arXiv)

This paper does not present a novel method. Instead, it delves into an essential, yet must-know baseline in light of the latest advancements in Generative Artificial Intelligence (GenAI): the utilization of GPT-4 for visual understanding. Our study centers on the evaluation of GPT-4's linguistic and visual capabilities in zero-shot visual recognition tasks: Firstly, we explore the potential of its generated rich textual descriptions across various categories to enhance recognition performance without any training. Secondly, we evaluate GPT-4's visual proficiency in directly recognizing diverse visual content. We conducted extensive experiments to systematically evaluate GPT-4's performance across images, videos, and point clouds, using 16 benchmark datasets to measure top-1 and top-5 accuracy. Our findings show that GPT-4, enhanced with rich linguistic descriptions, significantly improves zero-shot recognition, offering an average top-1 accuracy increase of 7% across all datasets. GPT-4 excels in visual recognition, outshining OpenAI-CLIP's ViT-L and rivaling EVA-CLIP's ViT-E, particularly in video datasets HMDB-51 and UCF-101, where it leads by 22% and 9%, respectively. We hope this research contributes valuable data points and experience for future studies. We release our code at https://github.com/whwu95/GPT4Vis.