mlsanityData to 7 Oct 2026

IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models

arXiv 2305.14985 ↗First posted 24 May 2023cs.CV cs.CLcited by 45

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

The field of vision-and-language (VL) understanding has made unprecedented progress with end-to-end large pre-trained VL models (VLMs). However, they still fall short in zero-shot reasoning tasks that require multi-step inferencing. To achieve this goal, previous works resort to a divide-and-conquer pipeline. In this paper, we argue that previous efforts have several inherent shortcomings: 1) They rely on domain-specific sub-question decomposing models. 2) They force models to predict the final answer even if the sub-questions or sub-answers provide insufficient information. We address these limitations via IdealGPT, a framework that iteratively decomposes VL reasoning using large language models (LLMs). Specifically, IdealGPT utilizes an LLM to generate sub-questions, a VLM to provide corresponding sub-answers, and another LLM to reason to achieve the final answer. These three modules perform the divide-and-conquer procedure iteratively until the model is confident about the final answer to the main question. We evaluate IdealGPT on multiple challenging VL reasoning tasks under a zero-shot setting. In particular, our IdealGPT outperforms the best existing GPT-4-like models by an absolute 10% on VCR and 15% on SNLI-VE. Code is available at https://github.com/Hxyou/IdealGPT

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 36 of its 51 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

36 of 51 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action20 Mar
  2. 2023ViperGPT: Visual Inference via Python Execution for Reasoning14 Mar
  3. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models8 Mar
  4. 2022Visual Programming: Compositional visual reasoning without training18 Nov

Other references · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2015VQA: Visual Question Answeringcited by 2,489
20 more
  1. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  2. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  3. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  4. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  5. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  6. 2023PaLM 2 Technical Reportcited by 1,154
  7. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  8. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  9. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  10. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  11. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  12. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  13. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  14. 2023Answering Questions by Meta-Reasoning over Multiple Chains of Thoughtcited by 87
  15. 2023ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptionscited by 86
  16. 2022Webly Supervised Concept Expansion for General Purpose Vision Modelscited by 45
  17. 2022Learning Visual Representation from Modality-Shared Contrastive Language-Image Pre-trainingcited by 32
  18. 2023CoBIT: A Contrastive Bi-directional Image-Text Generation Modelcited by 11
  19. 2022Co-VQA : Answering by Interactive Sub Question Sequencecited by 8
  20. 2022Understanding ME? Multimodal Evaluation for Fine-grained Visual Commonsensecited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 20; 2025: 8; 2026: 5.

45 papers.

Sort:

By area:

  1. 2023A Survey on Multimodal Large Language Modelscited by 898
  2. 2023A Comprehensive Overview of Large Language Modelscited by 639
  3. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  4. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  5. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 223
  6. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  7. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 136
  8. 2023VIGC: Visual Instruction Generation and Correctioncited by 80
  9. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 74
  10. 2024Visual Agents as Fast and Slow Thinkerscited by 53
  11. 2024Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoningcited by 37
  12. 2024HaloQuest: A Visual Hallucination Dataset for Advancing Multimodal Reasoningcited by 30
33 more
  1. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 26
  2. 2024ConvBench: A Multi-Turn Conversation Evaluation Benchmark with Hierarchical Capability for Large Vision-Language Modelscited by 24
  3. 2024An X-Ray Is Worth 15 Features: Sparse Autoencoders for Interpretable Radiology Report Generationcited by 23
  4. 2024AutoBench-V: Can Large Vision-Language Models Benchmark Themselves?cited by 19
  5. 2024LogiCode: an LLM-Driven Framework for Logical Anomaly Detectioncited by 18
  6. 2024HYDRA: A Hyper Agent for Dynamic Compositional Visual Reasoningcited by 17
  7. 2024MMCTAgent: Multi-modal Critical Thinking Agent Framework for Complex Visual Reasoningcited by 13
  8. 2024Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awarenesscited by 13
  9. 2023ViCor: Bridging Visual Understanding and Commonsense Reasoning with Large Language Modelscited by 12
  10. 2023Towards Top-Down Reasoning: An Explainable Multi-Agent Approach for Visual Question Answeringcited by 11
  11. 2024Modeling Collaborator: Enabling Subjective Vision Classification With Minimal Human Effort via LLM Tool-Usecited by 9
  12. 2024MOSABench: Multi-Object Sentiment Analysis Benchmark for Evaluating Multimodal Large Language Models Understanding of Complex Imagecited by 9
  13. 2023MLLM-DataEngine: An Iterative Refinement Approach for MLLMcited by 8
  14. 2023Good Questions Help Zero-Shot Image Reasoningcited by 8
  15. 2024NEUSIS: A Compositional Neuro-Symbolic Framework for Autonomous Perception, Reasoning, and Planning in Complex UAV Search Missionscited by 8
  16. 2023Tackling Vision Language Tasks Through Learning Inner Monologuescited by 6
  17. 2024Prompting Large Vision-Language Models for Compositional Reasoningcited by 6
  18. 2025DSADF: Thinking Fast and Slow for Decision Makingcited by 6
  19. 2024Generalizing Visual Question Answering from Synthetic to Human-Written Questions via a Chain of QA with a Large Language Modelcited by 5
  20. 2024Benchmarking VLMs' Reasoning About Persuasive Atypical Imagescited by 5
  21. 2024JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Imagescited by 5
  22. 2024CAP: Evaluation of Persuasive and Creative Image Generationcited by 5
  23. 2025JRDB-Reasoning: A Difficulty-Graded Benchmark for Visual Reasoning in Roboticscited by 3
  24. 2023Less is More: Toward Zero-Shot Local Scene Graph Generation via Foundation Modelscited by 1
  25. 2025LumiGen: An LVLM-Enhanced Iterative Framework for Fine-Grained Text-to-Image Generationcited by 1
  26. 2025Unbiased Visual Reasoning with Controlled Visual Inputscited by 1
  27. 2025A Multi-Modal Neuro-Symbolic Approach for Spatial Reasoning-Based Visual Grounding in Roboticscited by 0
  28. 2025MIND: Multi-rationale INtegrated Discriminative Reasoning Framework for Multi-modal Large Modelscited by 0
  29. 2026Decoding the Pulse of Reasoning VLMs in Multi-Image Understanding Taskscited by 0
  30. 2026Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generationcited by 0
  31. 2026MLLM-DataEngine: Closing the Loop of Multimodal Instruction Tuning Data Generationcited by 0
  32. 2026Visual Saliency Steering Distillation for Multimodal Chain-of-Thought Reasoningcited by 0
  33. 2026Structure-Token Evidence-Anchored Reasoning for Scientific Chart Understandingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Zero-Shot Visual Reasoning by Vision-Language Models: Benchmarking and Analysis
  2. 2025Elevating Visual Question Answering through Implicitly Learned Reasoning Pathways in LVLMs
  3. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks
  4. 2024FiVL: A Framework for Improved Vision-Language Alignment through the Lens of Training, Evaluation and Explainability
  5. 2025Socratic-MCTS: Test-Time Visual Reasoning by Asking the Right Questions
  6. 2023Improving Zero-shot Visual Question Answering via Large Language Models with Reasoning Question Prompts
  7. 2025VReST: Enhancing Reasoning in Large Vision-Language Models through Tree Search and Self-Reward Mechanism
  8. 2024Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs