mlsanityData to 14 Oct 2026

Image-of-Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Models

arXiv 2405.13872 ↗First posted 22 May 2024cs.AI cs.CL cs.CVcited by 65

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Recent advancements in Chain-of-Thought (CoT) and related rationale-based works have significantly improved the performance of Large Language Models (LLMs) in complex reasoning tasks. With the evolution of Multimodal Large Language Models (MLLMs), enhancing their capability to tackle complex multimodal reasoning problems is a crucial frontier. However, incorporating multimodal rationales in CoT has yet to be thoroughly investigated. We propose the Image-of-Thought (IoT) prompting method, which helps MLLMs to extract visual rationales step-by-step. Specifically, IoT prompting can automatically design critical visual information extraction operations based on the input images and questions. Each step of visual information refinement identifies specific visual rationales that support answers to complex visual reasoning questions. Beyond the textual CoT, IoT simultaneously utilizes visual and textual rationales to help MLLMs understand complex multimodal information. IoT prompting has improved zero-shot visual reasoning performance across various visual understanding tasks in different MLLMs. Moreover, the step-by-step visual feature explanations generated by IoT prompting elucidate the visual reasoning process, aiding in analyzing the cognitive processes of large multimodal models

Key facts. Cited by 65 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 39 of its 46 references that are in the map. First posted on arXiv 22 May 2024.

Builds on

39 of 46 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputs5 Jan
  2. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Models27 Nov
  3. 2023DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Models25 Oct
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  5. 2023Multimodal Chain-of-Thought Reasoning in Language Models2 Feb

Other references · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2023Segment Anythingcited by 8,234
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  11. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  12. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
22 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  3. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  5. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  6. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 688
  7. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  8. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 558
  9. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 388
  10. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 350
  11. 2023Explainability for Large Language Models: A Surveycited by 333
  12. 2022Large Language Models Are Reasoning Teacherscited by 329
  13. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  14. 2023Fast Segment Anythingcited by 290
  15. 2022Teaching Small Language Models to Reasoncited by 284
  16. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 101
  17. 2023Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animatorcited by 74
  18. 2024A Multimodal Automated Interpretability Agentcited by 63
  19. 2024DetToolChain: A New Prompting Paradigm to Unleash Detection Ability of MLLMcited by 46
  20. 2023CoTDet: Affordance Knowledge Prompting for Task Driven Object Detectioncited by 36
  21. 2024How Interpretable are Reasoning Explanations from Prompting Large Language Models?cited by 29
  22. 2024Joint Visual and Text Prompting for Improved Object-Centric Perception with Multimodal Large Language Modelscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 35; 2026: 23.

65 papers.

Sort:

By area:

  1. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 182
  2. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmarkcited by 134
  3. 2025Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokenscited by 124
  4. 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Modelscited by 95
  5. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  6. 2025Visual Planning: Let's Think Only with Imagescited by 85
  7. 2025Grounded Reinforcement Learning for Visual Reasoningcited by 83
  8. 2025PyVision: Agentic Vision with Dynamic Toolingcited by 79
  9. 2026Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillationcited by 77
  10. 2025ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoningcited by 56
  11. 2026Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perceptioncited by 50
  12. 2025Perspective-Aware Reasoning in Vision-Language Models via Mental Imagery Simulationcited by 42
38 more
  1. 2024EgoPlan-Bench2: A Benchmark for Multimodal Large Language Model Planning in Real-World Scenarioscited by 41
  2. 2025Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thoughtcited by 35
  3. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 33
  4. 2024What is the Visual Cognition Gap between Humans and Multimodal LLMs?cited by 30
  5. 2024Thinking Before Looking: Improving Multimodal LLM Reasoning via Mitigating Visual Hallucinationcited by 29
  6. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 28
  7. 2025Mind with Eyes: from Language Reasoning to Multimodal Reasoningcited by 22
  8. 2025Scaling and Beyond: Advancing Spatial Reasoning in MLLMs Requires New Recipescited by 22
  9. 2025RSVP: Reasoning Segmentation via Visual Prompting and Multi-modal Chain-of-Thoughtcited by 19
  10. 2025Mull-Tokens: Modality-Agnostic Latent Thinkingcited by 19
  11. 2025Thinking with Visual Abstract: Enhancing Multimodal Reasoning via Visual Abstractioncited by 15
  12. 2025Seeing and Reasoning with Confidence: Supercharging Multimodal LLMs with an Uncertainty-Aware Agentic Frameworkcited by 12
  13. 2025PixelCraft: A Multi-Agent System for High-Fidelity Visual Reasoning on Structured Imagescited by 12
  14. 2025Investigating Inference-time Scaling for Chain of Multi-modal Thought: A Preliminary Studycited by 11
  15. 2025ViPER: Empowering the Self-Evolution of Visual Perception Abilities in Vision-Language Modelcited by 11
  16. 2024A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Futurecited by 9
  17. 2025BLINK-Twice: You see, but do you observe? A Reasoning Benchmark on Visual Perceptioncited by 9
  18. 2025Octopus: Agentic Multimodal Reasoning with Six-Capability Orchestrationcited by 9
  19. 2025Mixture-of-Visual-Thoughts: Exploring Context-Adaptive Reasoning Mode Selection for General Visual Reasoningcited by 7
  20. 2025Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMscited by 7
  21. 2026Think in Strokes, Not Pixels: Process-Driven Image Generation via Interleaved Reasoningcited by 7
  22. 2024MageBench: Bridging Large Multimodal Models to Agentscited by 6
  23. 2025GoViG: Goal-Conditioned Visual Navigation Instruction Generation via Multimodal Reasoningcited by 6
  24. 2026Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMscited by 6
  25. 2024Autonomous Imagination: Closed-Loop Decomposition of Visual-to-Textual Conversion in Visual Reasoning for Multimodal Large Language Modelscited by 5
  26. 2025Introspection of Thought Helps AI Agentscited by 4
  27. 2025PRISM-Bench: A Benchmark of Puzzle-Based Visual Tasks with CoT Error Detectioncited by 4
  28. 2026Let's Think with Images Efficiently! An Interleaved-Modal Chain-of-Thought Reasoning Framework with Dynamic and Precise Visual Thoughtscited by 4
  29. 2026TABQAWORLD: Optimizing Multimodal Reasoning for Multi-Turn Table Question Answeringcited by 4
  30. 2024Image First or Text First? Optimising the Sequencing of Modalities in Large Language Model Prompting and Reasoning Taskscited by 3
  31. 2025On Data Synthesis and Post-training for Visual Abstract Reasoningcited by 3
  32. 2025RxnCaption: Reformulating Reaction Diagram Parsing as Visual Prompt Guided Captioningcited by 3
  33. 2025Agentic Design Review Systemcited by 2
  34. 2025Planning with Unified Multimodal Modelscited by 2
  35. 2026Molecular Identifier Visual Prompt and Verifiable Reinforcement Learning for Chemical Reaction Diagram Parsingcited by 2
  36. 2025From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Modelscited by 1
  37. 2026GeoFocus: Blending Efficient Global-to-Local Perception for Multimodal Geometry Problem-Solvingcited by 1
  38. 2026Enhancing MLLM Spatial Understanding via Active 3D Scene Exploration for Multi-Perspective Reasoningcited by 1

50 most cited of 65 All 65 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Imagine while Reasoning in Space: Multimodal Visualization-of-Thought
  2. 2026TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding
  3. 2026Attention-guided Fine-tuning of Multimodal Large Language Models Improves Chain-of-Thought Reasoning
  4. 2024Cantor: Inspiring Multimodal Chain-of-Thought of MLLM
  5. 2023See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning
  6. 2025VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoning
  7. 2025Argus: Vision-Centric Reasoning with Grounded Chain-of-Thought
  8. 2025QG-CoC: Question-Guided Chain-of-Captions for Large Multimodal Models