mlsanityData to 14 Oct 2026

Assessing GPT4-V on Structured Reasoning Tasks

arXiv 2312.11524 ↗First posted 13 Dec 2023cs.CL cs.AI cs.CVcited by 14

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Multi-modality promises to unlock further uses for large language models. Recently, the state-of-the-art language model GPT-4 was enhanced with vision capabilities. We carry out a prompting evaluation of GPT-4V and five other baselines on structured reasoning tasks, such as mathematical reasoning, visual data analysis, and code generation. We show that visual Chain-of-Thought, an extension of Chain-of-Thought to multi-modal LLMs, yields significant improvements over the vanilla model. We also present a categorized analysis of scenarios where these models perform well and where they struggle, highlighting challenges associated with coherent multimodal reasoning.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 9 of its 27 references that are in the map. First posted on arXiv 13 Dec 2023.

Builds on

9 of 27 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Multimodal Chain-of-Thought Reasoning in Language Models2 Feb

Other references · 8

  1. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  2. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  3. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  4. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  5. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  6. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 286
  7. 2019On the Measure of Intelligencecited by 144
  8. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125

Built on by

Cited by year

Citing papers per year: 2024: 8; 2025: 6.

14 papers.

Sort:

By area:

  1. 2024M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thoughtcited by 121
  2. 2024Are Large Vision Language Models up to the Challenge of Chart Comprehension and Reasoning? An Extensive Investigation into the Capabilities and Limitations of LVLMscited by 30
  3. 2025MMSciBench: Benchmarking Language Models on Chinese Multimodal Scientific Problemscited by 10
  4. 2024Exploring Boundary of GPT-4V on Marine Analysis: A Preliminary Case Studycited by 9
  5. 2025Score the Steps, Not Just the Goal: VLM-Based Subgoal Evaluation for Robotic Manipulationcited by 8
  6. 2024Program Synthesis using Inductive Logic Programming for the Abstraction and Reasoning Corpuscited by 4
  7. 2024Vision Language Models for Spreadsheet Understanding: Challenges and Opportunitiescited by 4
  8. 2024TurtleBench: A Visual Programming Benchmark in Turtle Geometrycited by 4
  9. 2024GeoBiked: A Dataset with Geometric Features and Automated Labeling Techniques to Enable Deep Generative Models in Engineering Designcited by 2
  10. 2024Meaningless is better: hashing bias-inducing words in LLM prompts improves performance in logical reasoning and statistical learningcited by 1
  11. 2025Vector Symbolic Algebras for the Abstraction and Reasoning Corpuscited by 1
  12. 2025AgroCoT: A Chain-of-Thought Benchmark for Evaluating Reasoning in Vision-Language Models for Agriculturecited by 1
2 more
  1. 2025MIND: Multi-rationale INtegrated Discriminative Reasoning Framework for Multi-modal Large Modelscited by 1
  2. 2025MarineEval: Assessing the Marine Intelligence of Vision-Language Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Models
  2. 2025Diagnosing Visual Reasoning: Challenges, Insights, and a Path Forward
  3. 2025VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoning
  4. 2024Cantor: Inspiring Multimodal Chain-of-Thought of MLLM
  5. 2025MM-CoT:A Benchmark for Probing Visual Chain-of-Thought Reasoning in Multimodal Models
  6. 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models
  7. 2026CoVA-SFT: A Large-Scale Dataset for Chain of Visual Abstractions
  8. 2026Look Light, Think Heavy: What Multimodal Chain-of-Thought Reasoning Can and Cannot Do