# CollagePrompt: A Benchmark for Budget-Friendly Visual Recognition with GPT-4V

arXiv: https://arxiv.org/abs/2403.11468 · HTML: https://mlsanity.com/paper/2403.11468/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 52 references that are in the map.
- First posted on arXiv 18 Mar 2024.

## Builds on (most cited first, 23 of 23)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models](https://mlsanity.com/paper/2303.04671/) (2023)
- [GPT-4V(ision) is a Generalist Web Agent, if Grounded](https://mlsanity.com/paper/2401.01614/) (2024)
- [A Survey on Multimodal Large Language Models for Autonomous Driving](https://mlsanity.com/paper/2311.12320/) (2023)
- [Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning](https://mlsanity.com/paper/2311.17842/) (2023)
- [Batch Prompting: Efficient Inference with Large Language Model APIs](https://mlsanity.com/paper/2301.08721/) (2023)
- [MM-VID: Advancing Video Understanding with GPT-4V(ision)](https://mlsanity.com/paper/2310.19773/) (2023)
- [Hierarchical Graph Pooling with Structure Learning](https://mlsanity.com/paper/1911.05954/) (2019)
- [On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Driving](https://mlsanity.com/paper/2311.05332/) (2023)
- [Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosis](https://mlsanity.com/paper/2310.09909/) (2023)
- [GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstration](https://mlsanity.com/paper/2311.12015/) (2023)
- [Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluation](https://mlsanity.com/paper/2310.16809/) (2023)
- [DME-Driver: Integrating Human Decision Logic and 3D Scene Perception in Autonomous Driving](https://mlsanity.com/paper/2401.03641/) (2024)
- [Evolutionary Generative Adversarial Networks](https://mlsanity.com/paper/1803.00657/) (2018)
- [A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysis](https://mlsanity.com/paper/2310.20381/) (2023)
- [GPT4Vis: What Can GPT-4 Do for Zero-shot Visual Recognition?](https://mlsanity.com/paper/2311.15732/) (2023)
- [An Early Evaluation of GPT-4V(ision)](https://mlsanity.com/paper/2310.16534/) (2023)

## Built on by (most cited first, 2 of 2)

- [NTSEBENCH: Cognitive Reasoning Benchmark for Vision Language Models](https://mlsanity.com/paper/2407.10380/) (2024)
- [AutoJudger: An Agent-Driven Framework for Efficient Benchmarking of MLLMs](https://mlsanity.com/paper/2505.21389/) (2025)

## Abstract (arXiv)

Recent advancements in generative AI have suggested that by taking visual prompts, GPT-4V can demonstrate significant proficiency in visual recognition tasks. Despite its impressive capabilities, the financial cost associated with GPT-4V's inference presents a substantial barrier to its wide use. To address this challenge, we propose a budget-friendly collage prompting task that collages multiple images into a single visual prompt and makes GPT-4V perform visual recognition on several images simultaneously, thereby reducing the cost. We collect a dataset of various collage prompts to assess its performance in GPT-4V's visual recognition. Our evaluations reveal several key findings: 1) Recognition accuracy varies with different positions in the collage. 2) Grouping images of the same category together leads to better visual recognition results. 3) Incorrect labels often come from adjacent images. These findings highlight the importance of image arrangement within collage prompt. To this end, we construct a benchmark called CollagePrompt, which offers a platform for designing collage prompt to achieve more cost-effective visual recognition with GPT-4V. A baseline method derived from genetic algorithms to optimize collage layouts is proposed and two metrics are introduced to measure the efficiency of the optimized collage prompt. Our benchmark enables researchers to better optimize collage prompts, thus making GPT-4V more cost-effective in visual recognition. The code and data are available at this project page https://collageprompting.github.io/.