# Assessing GPT4-V on Structured Reasoning Tasks

arXiv: https://arxiv.org/abs/2312.11524 · HTML: https://mlsanity.com/paper/2312.11524/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 9 of its 27 references that are in the map.
- First posted on arXiv 13 Dec 2023.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 9 of 9)

- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [Multimodal Chain-of-Thought Reasoning in Language Models](https://mlsanity.com/paper/2302.00923/) (2023)
- [NExT-GPT: Any-to-Any Multimodal LLM](https://mlsanity.com/paper/2309.05519/) (2023)
- [SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities](https://mlsanity.com/paper/2305.11000/) (2023)
- [SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models](https://mlsanity.com/paper/2311.07575/) (2023)
- [On the Measure of Intelligence](https://mlsanity.com/paper/1911.01547/) (2019)
- [BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions](https://mlsanity.com/paper/2308.09936/) (2023)

## Built on by (most cited first, 14 of 14)

- [M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought](https://mlsanity.com/paper/2405.16473/) (2024)
- [Are Large Vision Language Models up to the Challenge of Chart Comprehension and Reasoning? An Extensive Investigation into the Capabilities and Limitations of LVLMs](https://mlsanity.com/paper/2406.00257/) (2024)
- [MMSciBench: Benchmarking Language Models on Chinese Multimodal Scientific Problems](https://mlsanity.com/paper/2503.01891/) (2025)
- [Exploring Boundary of GPT-4V on Marine Analysis: A Preliminary Case Study](https://mlsanity.com/paper/2401.02147/) (2024)
- [Score the Steps, Not Just the Goal: VLM-Based Subgoal Evaluation for Robotic Manipulation](https://mlsanity.com/paper/2509.19524/) (2025)
- [Program Synthesis using Inductive Logic Programming for the Abstraction and Reasoning Corpus](https://mlsanity.com/paper/2405.06399/) (2024)
- [Vision Language Models for Spreadsheet Understanding: Challenges and Opportunities](https://mlsanity.com/paper/2405.16234/) (2024)
- [TurtleBench: A Visual Programming Benchmark in Turtle Geometry](https://mlsanity.com/paper/2411.00264/) (2024)
- [GeoBiked: A Dataset with Geometric Features and Automated Labeling Techniques to Enable Deep Generative Models in Engineering Design](https://mlsanity.com/paper/2409.17045/) (2024)
- [Meaningless is better: hashing bias-inducing words in LLM prompts improves performance in logical reasoning and statistical learning](https://mlsanity.com/paper/2411.17304/) (2024)
- [Vector Symbolic Algebras for the Abstraction and Reasoning Corpus](https://mlsanity.com/paper/2511.08747/) (2025)
- [AgroCoT: A Chain-of-Thought Benchmark for Evaluating Reasoning in Vision-Language Models for Agriculture](https://mlsanity.com/paper/2511.23253/) (2025)
- [MIND: Multi-rationale INtegrated Discriminative Reasoning Framework for Multi-modal Large Models](https://mlsanity.com/paper/2512.05530/) (2025)
- [MarineEval: Assessing the Marine Intelligence of Vision-Language Models](https://mlsanity.com/paper/2512.21126/) (2025)

## Abstract (arXiv)

Multi-modality promises to unlock further uses for large language models. Recently, the state-of-the-art language model GPT-4 was enhanced with vision capabilities. We carry out a prompting evaluation of GPT-4V and five other baselines on structured reasoning tasks, such as mathematical reasoning, visual data analysis, and code generation. We show that visual Chain-of-Thought, an extension of Chain-of-Thought to multi-modal LLMs, yields significant improvements over the vanilla model. We also present a categorized analysis of scenarios where these models perform well and where they struggle, highlighting challenges associated with coherent multimodal reasoning.