mlsanityData to 7 Oct 2026

Compositional Chain-of-Thought Prompting for Large Multimodal Models

arXiv 2311.17076 ↗First posted 27 Nov 2023cs.CV cs.AI cs.CLcited by 160

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

The combination of strong visual backbones and Large Language Model (LLM) reasoning has led to Large Multimodal Models (LMMs) becoming the current standard for a wide range of vision and language (VL) tasks. However, recent research has shown that even the most advanced LMMs still struggle to capture aspects of compositional visual reasoning, such as attributes and relationships between objects. One solution is to utilize scene graphs (SGs)--a formalization of objects and their relations and attributes that has been extensively used as a bridge between the visual and textual domains. Yet, scene graph data requires scene graph annotations, which are expensive to collect and thus not easily scalable. Moreover, finetuning an LMM based on SG data can lead to catastrophic forgetting of the pretraining objective. To overcome this, inspired by chain-of-thought methods, we propose Compositional Chain-of-Thought (CCoT), a novel zero-shot Chain-of-Thought prompting method that utilizes SG representations in order to extract compositional knowledge from an LMM. Specifically, we first generate an SG using the LMM, and then use that SG in the prompt to produce a response. Through extensive experiments, we find that the proposed CCoT approach not only improves LMM performance on several vision and language VL compositional benchmarks but also improves the performance of several popular LMMs on general multimodal benchmarks, without the need for fine-tuning or annotated ground-truth SGs. Code: https://github.com/chancharikmitra/CCoT

Key facts. Cited by 160 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 81 of its 93 references that are in the map. First posted on arXiv 27 Nov 2023.

Builds on

81 of 93 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Models25 Oct
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  3. 2023T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Mixed Large Language Model Signals for Science Question Answering5 May
  4. 2023ViperGPT: Visual Inference via Python Execution for Reasoning14 Mar
  5. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models8 Mar
  6. 2023Multimodal Chain-of-Thought Reasoning in Language Models2 Feb
  7. 2022Visual Programming: Compositional visual reasoning without training18 Nov
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep
  9. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality7 Apr

Other references · 72

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
38 more
  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2015VQA: Visual Question Answeringcited by 2,489
  13. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  14. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  15. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  16. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  17. 2018Relational inductive biases, deep learning, and graph networkscited by 1,341
  18. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  19. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  20. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  21. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  22. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  23. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  24. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  25. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  26. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
  27. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  28. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  29. 2015Neural Module Networkscited by 375
  30. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  31. 2018Image Generation from Scene Graphscited by 324
  32. 2022UL2: Unifying Language Learning Paradigmscited by 292
  33. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  34. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  35. 2018Videos as Space-Time Region Graphscited by 221
  36. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  37. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  38. 2023SVIT: Scaling up Visual Instruction Tuningcited by 161

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 48; 2025: 68; 2026: 44.

160 papers.

Sort:

By area:

  1. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 653
  2. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
  3. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  4. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 171
  5. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 144
  6. 2025GRIT: Teaching MLLMs to Think with Imagescited by 121
  7. 2024M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thoughtcited by 120
  8. 2024Large Language Models Meet NLP: A Surveycited by 102
  9. 2025Zebra-CoT: A Dataset for Interleaved Vision Language Reasoningcited by 78
  10. 2024Image-of-Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Modelscited by 64
  11. 2024CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputscited by 56
  12. 2024Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Modelscited by 56
38 more
  1. 2024What Makes Multimodal In-Context Learning Work?cited by 54
  2. 2025Uni-cot: Towards Unified Chain-of-Thought Reasoning Across Text and Visioncited by 52
  3. 2025Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learningcited by 51
  4. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learningcited by 49
  5. 2024Interleaved-Modal Chain-of-Thoughtcited by 47
  6. 2025DyFo: A Training-Free Dynamic Focus Visual Search for Enhancing LMMs in Fine-Grained Visual Understandingcited by 47
  7. 2024DetToolChain: A New Prompting Paradigm to Unleash Detection Ability of MLLMcited by 45
  8. 2025Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Rewardcited by 45
  9. 2024AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruningcited by 43
  10. 2025VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learningcited by 41
  11. 2024Core Knowledge Deficits in Multi-Modal Language Modelscited by 38
  12. 2024TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understandingcited by 37
  13. 2024EgoPlan-Bench2: A Benchmark for Multimodal Large Language Model Planning in Real-World Scenarioscited by 37
  14. 2024Enhancing Video-LLM Reasoning via Agent-of-Thoughts Distillationcited by 36
  15. 2024Cantor: Inspiring Multimodal Chain-of-Thought of MLLMcited by 34
  16. 2025A Survey on Mechanistic Interpretability for Multi-Modal Foundation Modelscited by 34
  17. 2025VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Searchcited by 34
  18. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulationcited by 34
  19. 2025Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thoughtcited by 34
  20. 2024Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrievalcited by 31
  21. 2024SketchAgent: Language-Driven Sequential Sketch Generationcited by 30
  22. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 26
  23. 2024TraveLER: A Modular Multi-LMM Agent Framework for Video Question-Answeringcited by 24
  24. 2025Towards Training-free Anomaly Detection with Vision and Language Foundation Modelscited by 24
  25. 2025Critique of World Modelcited by 24
  26. 2025Mind with Eyes: from Language Reasoning to Multimodal Reasoningcited by 22
  27. 2024MC-CoT: A Modular Collaborative CoT Framework for Zero-shot Medical-VQA with LLM and MLLM Integrationcited by 21
  28. 2025Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Modelscited by 21
  29. 2024Understanding Figurative Meaning through Explainable Visual Entailmentcited by 19
  30. 2024Order Matters: Exploring Order Sensitivity in Multimodal Large Language Modelscited by 19
  31. 2025Aligning Vision to Language: Annotation-Free Multimodal Knowledge Graph Construction for Enhanced LLMs Reasoningcited by 19
  32. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  33. 2024Describe-then-Reason: Improving Multimodal Mathematical Reasoning through Visual Comprehension Trainingcited by 17
  34. 2024Large Language Models for Mobility Analysis in Transportation Systems: A Survey on Forecasting Taskscited by 15
  35. 2024SILMM: Self-Improving Large Multimodal Models for Compositional Text-to-Image Generationcited by 15
  36. 2025A Unified Agentic Framework for Evaluating Conditional Image Generationcited by 15
  37. 2024CoDA: Instructive Chain-of-Domain Adaptation with Severity-Aware Visual Prompt Tuningcited by 13
  38. 2025STOP: Integrated Spatial-Temporal Dynamic Prompting for Video Understandingcited by 13

50 most cited of 160 All 160 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoning
  2. 2026Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models
  3. 2025Evaluating Compositional Scene Understanding in Multimodal Generative Models
  4. 2025Grounded Chain-of-Thought for Multimodal Large Language Models
  5. 2025Understand, Think, and Answer: Advancing Visual Reasoning with Large Multimodal Models
  6. 2025Multimodal Chain of Continuous Thought for Latent-Space Reasoning in Vision-Language Models
  7. 2025QG-CoC: Question-Guided Chain-of-Captions for Large Multimodal Models
  8. 2026CoVA-SFT: A Large-Scale Dataset for Chain of Visual Abstractions