mlsanityData to 14 Oct 2026

Visual Chain of Thought: Bridging Logical Gaps with Multimodal Infillings

arXiv 2305.02317 ↗First posted 3 May 2023cs.CL cs.CVcited by 69

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Recent advances in large language models elicit reasoning in a chain-of-thought that allows models to decompose problems in a human-like fashion. Though this paradigm improves multi-step reasoning ability in language models, it is limited by being unimodal and applied mainly to question-answering tasks. We claim that incorporating visual augmentation into reasoning is essential, especially for complex, imaginative tasks. Consequently, we introduce VCoT, a novel method that leverages chain-of-thought prompting with vision-language grounding to recursively bridge the logical gaps within sequential data. Our method uses visual guidance to generate synthetic multimodal infillings that add consistent and novel information to reduce the logical gaps for downstream tasks that can benefit from temporal reasoning, as well as provide interpretability into models' multi-step reasoning. We apply VCoT to the Visual Storytelling and WikiHow summarization datasets and demonstrate through human evaluation that VCoT offers novel and consistent synthetic data augmentation beating chain-of-thought baselines, which can be used to enhance downstream performance.

Key facts. Cited by 69 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 27 of its 38 references that are in the map. First posted on arXiv 3 May 2023.

Builds on

27 of 38 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  8. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  9. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  10. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  11. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  12. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
15 more
  1. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  2. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
  3. 2022Can language models learn from explanations in context?cited by 249
  4. 2018WikiHow: A Large Scale Text Summarization Datasetcited by 204
  5. 2016Visual Storytellingcited by 176
  6. 2017A Knowledge-Grounded Neural Conversation Modelcited by 136
  7. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  8. 2022Rationale-Augmented Ensembles in Language Modelscited by 111
  9. 2020Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervisioncited by 94
  10. 2020Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentationcited by 42
  11. 2020DialogBERT: Discourse-Aware Response Generation via Learning to Recover and Rank Utterancescited by 36
  12. 2022Visualize Before You Write: Imagination-Guided Open-Ended Text Generationcited by 30
  13. 2022Imagination-Augmented Natural Language Understandingcited by 19
  14. 2019A Hybrid Retrieval-Generation Neural Conversation Modelcited by 18
  15. 2021SocAoG: Incremental Graph Parsing for Social Relation Inference in Dialoguescited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 15; 2025: 31; 2026: 14.

69 papers.

Sort:

By area:

  1. 2023A Survey on Multimodal Large Language Modelscited by 909
  2. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 451
  3. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
  4. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 182
  5. 2025WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agentcited by 119
  6. 2024Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Modelscited by 113
  7. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  8. 2023LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluationcited by 87
  9. 2025Zebra-CoT: A Dataset for Interleaved Vision Language Reasoningcited by 84
  10. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 76
  11. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
  12. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
38 more
  1. 2025VLM-R3: Region Recognition, Reasoning, and Refinement for Enhanced Multimodal Chain-of-Thoughtcited by 49
  2. 2025Perspective-Aware Reasoning in Vision-Language Models via Mental Imagery Simulationcited by 42
  3. 2024VideoCoT: A Video Chain-of-Thought Dataset with Active Annotation Toolcited by 35
  4. 2024EvoChart: A Benchmark and a Self-Training Approach Towards Real-World Chart Understandingcited by 34
  5. 2025Point-RFT: Improving Multimodal Reasoning with Visually Grounded Reinforcement Finetuningcited by 32
  6. 2024SketchAgent: Language-Driven Sequential Sketch Generationcited by 31
  7. 2025Can Large Vision Language Models Read Maps Like a Human?cited by 30
  8. 2025Argus: Vision-Centric Reasoning with Grounded Chain-of-Thoughtcited by 27
  9. 2025ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMscited by 25
  10. 2025MEDDxAgent: A Unified Modular Agent Framework for Explainable Automatic Differential Diagnosiscited by 23
  11. 2023The Art of SOCRATIC QUESTIONING: Recursive Thinking with Large Language Modelscited by 22
  12. 2026Thinking with Blueprints: Assisting Vision-Language Models in Spatial Reasoning via Structured Object Representationcited by 21
  13. 2026Visual-OPSD: Cross-Modal On-Policy Self-Distillation for Efficient Unified Multimodal Reasoningcited by 16
  14. 2024Routing Experts: Learning to Route Dynamic Experts in Multi-modal Large Language Modelscited by 14
  15. 2024CoDA: Instructive Chain-of-Domain Adaptation with Severity-Aware Visual Prompt Tuningcited by 13
  16. 2026Generative Visual Code Mobile World Modelscited by 13
  17. 2024Propose, Assess, Search: Harnessing LLMs for Goal-Oriented Planning in Instructional Videoscited by 12
  18. 2025Interactive Sketchpad: A Multimodal Tutoring System for Collaborative, Visual Problem-Solvingcited by 10
  19. 2024A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Futurecited by 9
  20. 2023V-LoL: A Diagnostic Dataset for Visual Logical Learningcited by 8
  21. 2024VACoDe: Visual Augmented Contrastive Decodingcited by 8
  22. 2025LaRe: Latent Refocusing for Multimodal Reasoningcited by 8
  23. 2024Soft-Prompting with Graph-of-Thought for Multi-modal Representation Learningcited by 7
  24. 2025GRADEO: Towards Human-Like Evaluation for Text-to-Video Generation via Multi-Step Reasoningcited by 7
  25. 2025StreamingCoT: A Dataset for Temporal Dynamics and Multimodal Chain-of-Thought Reasoning in Streaming VideoQAcited by 6
  26. 2023Let's Think Frame by Frame with VIP: A Video Infilling and Prediction Dataset for Evaluating Video Chain-of-Thoughtcited by 5
  27. 2025VITED: Video Temporal Evidence Distillationcited by 5
  28. 2025ReFineVLA: Reasoning-Aware Teacher-Guided Transfer Fine-Tuningcited by 5
  29. 2025Less is More Tokens: Efficient Math Reasoning via Difficulty-Aware Chain-of-Thought Distillationcited by 4
  30. 2026A Progressive Training Strategy for Embodied Vision-Language Models to Mitigate Spatio-Temporal Hallucinationscited by 4
  31. 2026Unveiling Fine-Grained Visual Traces: Evaluating Multimodal Interleaved Reasoning Chains in Multimodal STEM Taskscited by 4
  32. 2025Nature's Insight: A Novel Framework and Comprehensive Analysis of Agentic Reasoning Through the Lens of Neurosciencecited by 3
  33. 2025Cognitive Chain-of-Thought (CoCoT): Structured Multimodal Reasoning about Social Situationscited by 3
  34. 2025MEENA (PersianMMMU): Multimodal-Multilingual Educational Exams for N-level Assessmentcited by 3
  35. 2025SeeingEye: Agentic Information Flow Unlocks Multimodal Reasoning In Text-only LLMscited by 3
  36. 2025When to Think and When to Look: Uncertainty-Guided Lookbackcited by 3
  37. 2025Think Before You Move: Latent Motion Reasoning for Text-to-Motion Generationcited by 3
  38. 2023Training on Synthetic Data Beats Real Data in Multimodal Relation Extractioncited by 2

50 most cited of 69 All 69 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoning
  2. 2026Gen-VCoT: Generative Visual Chain-of-Thought Reasoning via Diffusion-Based RGB Intermediate Representations
  3. 2025Video Finetuning Improves Reasoning Between Frames
  4. 2025Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought
  5. 2026Genesis: Visual Augmentation Primitives for Multimodal Chain-of-Thought Reasoning
  6. 2026TwiFF (Think With Future Frames): A Large-Scale Dataset for Dynamic Visual Reasoning
  7. 2025Retrieval-Based Interleaved Visual Chain-of-Thought in Real-World Driving Scenarios
  8. 2025Multimodal Chain of Continuous Thought for Latent-Space Reasoning in Vision-Language Models