# Chain of Thought Prompt Tuning in Vision Language Models

arXiv: https://arxiv.org/abs/2304.07919 · HTML: https://mlsanity.com/paper/2304.07919/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 8 of its 43 references that are in the map.
- First posted on arXiv 16 Apr 2023.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 8 of 8)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [Prompt-aligned Gradient for Prompt Tuning](https://mlsanity.com/paper/2205.14865/) (2022)
- [How Much Can CLIP Benefit Vision-and-Language Tasks?](https://mlsanity.com/paper/2107.06383/) (2021)
- [Language models show human-like content effects on reasoning tasks](https://mlsanity.com/paper/2207.07051/) (2022)

## Built on by (most cited first, 30 of 34)

- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [A Survey on Multimodal Large Language Models for Autonomous Driving](https://mlsanity.com/paper/2311.12320/) (2023)
- [Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey](https://mlsanity.com/paper/2503.12605/) (2025)
- [AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Prompting](https://mlsanity.com/paper/2403.09513/) (2024)
- [Fine-tuning can cripple your foundation model; preserving features may be the solution](https://mlsanity.com/paper/2308.13320/) (2023)
- [Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models](https://mlsanity.com/paper/2505.04921/) (2025)
- [Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Character](https://mlsanity.com/paper/2405.20773/) (2024)
- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Models](https://mlsanity.com/paper/2312.02896/) (2023)
- [Autoregressive Models in Vision: A Survey](https://mlsanity.com/paper/2411.05902/) (2024)
- [Visual Large Language Models for Generalized and Specialized Applications](https://mlsanity.com/paper/2501.02765/) (2025)
- [Argus: Vision-Centric Reasoning with Grounded Chain-of-Thought](https://mlsanity.com/paper/2505.23766/) (2025)
- [Moderating Illicit Online Image Promotion for Unsafe User-Generated Content Games Using Large Vision-Language Models](https://mlsanity.com/paper/2403.18957/) (2024)
- [CoDA: Instructive Chain-of-Domain Adaptation with Severity-Aware Visual Prompt Tuning](https://mlsanity.com/paper/2403.17369/) (2024)
- [Reflective Instruction Tuning: Mitigating Hallucinations in Large Vision-Language Models](https://mlsanity.com/paper/2407.11422/) (2024)
- [A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Future](https://mlsanity.com/paper/2412.14056/) (2024)
- [VEGA: Learning Interleaved Image-Text Comprehension in Vision-Language Large Models](https://mlsanity.com/paper/2406.10228/) (2024)
- [Benchmarking XAI Explanations with Human-Aligned Evaluations](https://mlsanity.com/paper/2411.02470/) (2024)
- [Masked Diffusion with Task-awareness for Procedure Planning in Instructional Videos](https://mlsanity.com/paper/2309.07409/) (2023)
- [Soft-Prompting with Graph-of-Thought for Multi-modal Representation Learning](https://mlsanity.com/paper/2404.04538/) (2024)
- [Harnessing Chain-of-Thought Reasoning in Multimodal Large Language Models for Face Anti-Spoofing](https://mlsanity.com/paper/2506.01783/) (2025)
- [A Training-Free Framework for Video License Plate Tracking and Recognition with Only One-Shot](https://mlsanity.com/paper/2408.05729/) (2024)
- [XiHeFusion: Harnessing Large Language Models for Science Communication in Nuclear Fusion](https://mlsanity.com/paper/2502.05615/) (2025)
- [Multi-scale hypergraph meets LLMs: Aligning large language models for time series analysis](https://mlsanity.com/paper/2602.04369/) (2026)
- [Gradient constrained sharpness-aware prompt learning for vision-language models](https://mlsanity.com/paper/2309.07866/) (2023)
- [CPA-Enhancer: Chain-of-Thought Prompted Adaptive Enhancer for Object Detection under Unknown Degradations](https://mlsanity.com/paper/2403.11220/) (2024)
- [Trustworthy Large Models in Vision: A Survey](https://mlsanity.com/paper/2311.09680/) (2023)
- [Salience Adjustment for Context-Based Emotion Recognition](https://mlsanity.com/paper/2507.15878/) (2025)
- [Q-Doc: Benchmarking Document Image Quality Assessment Capabilities in Multi-modal Large Language Models](https://mlsanity.com/paper/2511.11410/) (2025)

## Abstract (arXiv)

Language-Image Pre-training has demonstrated promising results on zero-shot and few-shot downstream tasks by prompting visual models with natural language prompts. However, most recent studies only use a single prompt for tuning, neglecting the inherent step-to-step cognitive reasoning process that humans conduct in complex task settings, for example, when processing images from unfamiliar domains. Chain of Thought is a simple and effective approximation to human reasoning process and has been proven useful for natural language processing (NLP) tasks. Based on this cognitive intuition, we believe that conducting effective reasoning is also an important problem in visual tasks, and a chain of thought could be a solution to this problem. In this work, we propose a novel chain of thought prompt tuning for vision-language modeling. Extensive experiments show that our method not only generalizes better in image classification tasks, has greater transferability beyond a single dataset, and has stronger domain generalization performance, but also performs much better in imagetext retrieval and visual question answering, which require more reasoning capabilities. We are the first to successfully adapt chain-of-thought prompting that combines visual and textual embeddings. We will release our codes