mlsanityData to 14 Oct 2026

DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Models

arXiv 2310.16436 ↗First posted 25 Oct 2023cs.CV cs.CLcited by 128

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

A long-standing goal of AI systems is to perform complex multimodal reasoning like humans. Recently, large language models (LLMs) have made remarkable strides in such multi-step reasoning on the language modality solely by leveraging the chain of thought (CoT) to mimic human thinking. However, the transfer of these advancements to multimodal contexts introduces heightened challenges, including but not limited to the impractical need for labor-intensive annotation and the limitations in terms of flexibility, generalizability, and explainability. To evoke CoT reasoning in multimodality, this work first conducts an in-depth analysis of these challenges posed by multimodality and presents two key insights: "keeping critical thinking" and "letting everyone do their jobs" in multimodal CoT reasoning. Furthermore, this study proposes a novel DDCoT prompting that maintains a critical attitude through negative-space prompting and incorporates multimodality into reasoning by first dividing the reasoning responsibility of LLMs into reasoning and recognition and then integrating the visual recognition capability of visual models into the joint reasoning process. The rationales generated by DDCoT not only improve the reasoning abilities of both large and small language models in zero-shot prompting and fine-tuning learning, significantly outperforming state-of-the-art methods but also exhibit impressive generalizability and explainability.

Key facts. Cited by 128 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 65 of its 75 references that are in the map. First posted on arXiv 25 Oct 2023.

Builds on

65 of 75 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model28 Apr
  2. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models8 Mar
  3. 2023Multimodal Chain-of-Thought Reasoning in Language Models2 Feb
  4. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep

Other references · 61

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  8. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  9. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  10. 2023Visual Instruction Tuningcited by 7,869
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  12. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
38 more
  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  6. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  7. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  9. 2019Language Models as Knowledge Bases?cited by 1,936
  10. 2022Visual Prompt Tuningcited by 1,681
  11. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  12. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  13. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  14. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  15. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  16. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  17. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  18. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  19. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 719
  20. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  21. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 688
  22. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  23. 2022PAL: Program-aided Language Modelscited by 652
  24. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  25. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 558
  26. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  27. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 476
  28. 2018nocaps: novel object captioning at scalecited by 444
  29. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  30. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 388
  31. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356
  32. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 350
  33. 2017Six Challenges for Neural Machine Translationcited by 335
  34. 2022Large Language Models Are Reasoning Teacherscited by 329
  35. 2015Skip-Thought Vectorscited by 322
  36. 2022Teaching Small Language Models to Reasoncited by 284
  37. 2022Prompt Distribution Learningcited by 235
  38. 2020Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgettingcited by 170

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 44; 2025: 51; 2026: 31.

128 papers.

Sort:

By area:

  1. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 280
  2. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 182
  3. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 164
  4. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 145
  5. 2024Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Modelscited by 133
  6. 2024M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thoughtcited by 121
  7. 2024Large Multimodal Agents: A Surveycited by 105
  8. 2025VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuningcited by 93
  9. 2024Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Charactercited by 68
  10. 2024Towards Long-Horizon Vision-Language Navigation: Platform, Benchmark and Methodcited by 67
  11. 2024Image-of-Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Modelscited by 65
  12. 2025A Survey on (M)LLM-Based GUI Agentscited by 62
38 more
  1. 2024MMFakeBench: A Mixed-Source Multimodal Misinformation Detection Benchmark for LVLMscited by 61
  2. 2024CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputscited by 56
  3. 2025Uni-cot: Towards Unified Chain-of-Thought Reasoning Across Text and Visioncited by 56
  4. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  5. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  6. 2024Interleaved-Modal Chain-of-Thoughtcited by 51
  7. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learningcited by 50
  8. 2024Beyond Chain-of-Thought: A Survey of Chain-of-X Paradigms for LLMscited by 48
  9. 2024DetToolChain: A New Prompting Paradigm to Unleash Detection Ability of MLLMcited by 46
  10. 2024MedCoT: Medical Chain of Thought via Hierarchical Expertcited by 46
  11. 2025Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Rewardcited by 45
  12. 2025Step-GUI Technical Reportcited by 39
  13. 2024TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understandingcited by 38
  14. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulationcited by 35
  15. 2025Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thoughtcited by 35
  16. 2024Cantor: Inspiring Multimodal Chain-of-Thought of MLLMcited by 34
  17. 2024Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrievalcited by 32
  18. 2024What Factors Affect Multi-Modal In-Context Learning? An In-Depth Explorationcited by 31
  19. 2025HM-RAG: Hierarchical Multi-Agent Multimodal Retrieval Augmented Generationcited by 31
  20. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 28
  21. 2025Eyes Wide Open: Ego Proactive Video-LLM for Streaming Videocited by 28
  22. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  23. 2025A Comprehensive Survey on the Trustworthiness of Large Language Models in Healthcarecited by 27
  24. 2024Interactive Text-to-Image Retrieval with Large Language Models: A Plug-and-Play Approachcited by 25
  25. 2024An X-Ray Is Worth 15 Features: Sparse Autoencoders for Interpretable Radiology Report Generationcited by 25
  26. 2025Mind with Eyes: from Language Reasoning to Multimodal Reasoningcited by 22
  27. 2024MC-CoT: A Modular Collaborative CoT Framework for Zero-shot Medical-VQA with LLM and MLLM Integrationcited by 21
  28. 2025Can Multimodal Large Language Models Understand Spatial Relations?cited by 20
  29. 2025Aligning Vision to Language: Annotation-Free Multimodal Knowledge Graph Construction for Enhanced LLMs Reasoningcited by 19
  30. 2024Causal Evaluation of Language Modelscited by 18
  31. 2024Leopard: A Vision Language Model For Text-Rich Multi-Image Taskscited by 18
  32. 2025Vision Function Layer in Multimodal LLMscited by 17
  33. 2024Evaluating Fairness in Large Vision-Language Models Across Diverse Demographic Attributes and Promptscited by 15
  34. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 15
  35. 2024EEE-Bench: A Comprehensive Multimodal Electrical And Electronics Engineering Benchmarkcited by 15
  36. 2025Rethinking Query-based Transformer for Continual Image Segmentationcited by 14
  37. 2026ThinkRL-Edit: Thinking in Reinforcement Learning for Reasoning-Centric Image Editingcited by 12
  38. 2025A Causality-aware Paradigm for Evaluating Creativity of Multimodal Large Language Modelscited by 10

50 most cited of 128 All 128 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Look Light, Think Heavy: What Multimodal Chain-of-Thought Reasoning Can and Cannot Do
  2. 2025From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Models
  3. 2026Attention-guided Fine-tuning of Multimodal Large Language Models Improves Chain-of-Thought Reasoning
  4. 2026TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding
  5. 2024KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoning
  6. 2025Multimodal Chain of Continuous Thought for Latent-Space Reasoning in Vision-Language Models
  7. 2025Cognitive Chain-of-Thought (CoCoT): Structured Multimodal Reasoning about Social Situations
  8. 2023Towards Better Chain-of-Thought Prompting Strategies: A Survey