# CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputs

arXiv: https://arxiv.org/abs/2401.02582 · HTML: https://mlsanity.com/paper/2401.02582/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 17 of its 23 references that are in the map.
- First posted on arXiv 5 Jan 2024.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 17 of 17)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [Language Is Not All You Need: Aligning Perception with Language Models](https://mlsanity.com/paper/2302.14045/) (2023)
- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](https://mlsanity.com/paper/2204.03162/) (2022)
- [RAVEN: A Dataset for Relational and Analogical Visual rEasoNing](https://mlsanity.com/paper/1903.02741/) (2019)
- [MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning](https://mlsanity.com/paper/2309.07915/) (2023)
- [Compositional Chain-of-Thought Prompting for Large Multimodal Models](https://mlsanity.com/paper/2311.17076/) (2023)
- [Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learners](https://mlsanity.com/paper/2303.02151/) (2023)
- [DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Models](https://mlsanity.com/paper/2310.16436/) (2023)
- [PromptCap: Prompt-Guided Task-Aware Image Captioning](https://mlsanity.com/paper/2211.09699/) (2022)
- [Factify 2: A Multimodal Fake News and Satire News Dataset](https://mlsanity.com/paper/2304.03897/) (2023)
- [Unbiased Multi-Modality Guidance for Image Inpainting](https://mlsanity.com/paper/2208.11844/) (2022)
- [LLMVA-GEBC: Large Language Model with Video Adapter for Generic Event Boundary Captioning](https://mlsanity.com/paper/2306.10354/) (2023)

## Built on by (most cited first, 30 of 56)

- [Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey](https://mlsanity.com/paper/2503.12605/) (2025)
- [Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models](https://mlsanity.com/paper/2505.04921/) (2025)
- [Image-of-Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Models](https://mlsanity.com/paper/2405.13872/) (2024)
- [A Survey on (M)LLM-Based GUI Agents](https://mlsanity.com/paper/2504.13865/) (2025)
- [MMFakeBench: A Mixed-Source Multimodal Misinformation Detection Benchmark for LVLMs](https://mlsanity.com/paper/2406.08772/) (2024)
- [Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMs](https://mlsanity.com/paper/2502.19411/) (2025)
- [Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Reward](https://mlsanity.com/paper/2506.07218/) (2025)
- [TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understanding](https://mlsanity.com/paper/2404.09797/) (2024)
- [Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought](https://mlsanity.com/paper/2505.15510/) (2025)
- [Cantor: Inspiring Multimodal Chain-of-Thought of MLLM](https://mlsanity.com/paper/2404.16033/) (2024)
- [Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2412.11077/) (2024)
- [What is the Visual Cognition Gap between Humans and Multimodal LLMs?](https://mlsanity.com/paper/2406.10424/) (2024)
- [CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Models](https://mlsanity.com/paper/2412.12932/) (2024)
- [Thinking Before Looking: Improving Multimodal LLM Reasoning via Mitigating Visual Hallucination](https://mlsanity.com/paper/2411.12591/) (2024)
- [WeThink: Toward General-purpose Vision-Language Reasoning via Reinforcement Learning](https://mlsanity.com/paper/2506.07905/) (2025)
- [An Empirical Analysis on Large Language Models in Debate Evaluation](https://mlsanity.com/paper/2406.00050/) (2024)
- [Can Multimodal Large Language Models Understand Spatial Relations?](https://mlsanity.com/paper/2505.19015/) (2025)
- [Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding](https://mlsanity.com/paper/2403.16276/) (2024)
- [FINEMATCH: Aspect-based Fine-grained Image and Text Mismatch Detection and Correction](https://mlsanity.com/paper/2404.14715/) (2024)
- [Img-Diff: Contrastive Data Synthesis for Multimodal Large Language Models](https://mlsanity.com/paper/2408.04594/) (2024)
- [EAGLE: Egocentric AGgregated Language-video Engine](https://mlsanity.com/paper/2409.17523/) (2024)
- [IW-Bench: Evaluating Large Multimodal Models for Converting Image-to-Web](https://mlsanity.com/paper/2409.18980/) (2024)
- [Visual Reasoning Tracer: Object-Level Grounded Reasoning Benchmark](https://mlsanity.com/paper/2512.05091/) (2025)
- [Benchmarking Multimodal CoT Reward Model Stepwise by Visual Program](https://mlsanity.com/paper/2504.06606/) (2025)
- [Look Less, Reason More: Rollout-Guided Adaptive Pixel-Space Reasoning](https://mlsanity.com/paper/2510.01681/) (2025)
- [Natural Language Inference Improves Compositionality in Vision-Language Models](https://mlsanity.com/paper/2410.22315/) (2024)
- [MineAgent: Towards Remote-Sensing Mineral Exploration with Multimodal Large Language Models](https://mlsanity.com/paper/2412.17339/) (2024)
- [VISTA: Mitigating Semantic Inertia in Video-LLMs via Training-Free Dynamic Chain-of-Thought Routing](https://mlsanity.com/paper/2505.11830/) (2025)
- [X-Reflect: Cross-Reflection Prompting for Multimodal Recommendation](https://mlsanity.com/paper/2408.15172/) (2024)
- [Chain-of-Thought in Large Language Models: Decoding, Projection, and Activation](https://mlsanity.com/paper/2412.03944/) (2024)

## Abstract (arXiv)

When exploring the development of Artificial General Intelligence (AGI), a critical task for these models involves interpreting and processing information from multiple image inputs. However, Large Multimodal Models (LMMs) encounter two issues in such scenarios: (1) a lack of fine-grained perception, and (2) a tendency to blend information across multiple images. We first extensively investigate the capability of LMMs to perceive fine-grained visual details when dealing with multiple input images. The research focuses on two aspects: first, image-to-image matching (to evaluate whether LMMs can effectively reason and pair relevant images), and second, multi-image-to-text matching (to assess whether LMMs can accurately capture and summarize detailed image information). We conduct evaluations on a range of both open-source and closed-source large models, including GPT-4V, Gemini, OpenFlamingo, and MMICL. To enhance model performance, we further develop a Contrastive Chain-of-Thought (CoCoT) prompting approach based on multi-input multimodal models. This method requires LMMs to compare the similarities and differences among multiple image inputs, and then guide the models to answer detailed questions about multi-image inputs based on the identified similarities and differences. Our experimental results showcase CoCoT's proficiency in enhancing the multi-image comprehension capabilities of large multimodal models.