mlsanityData to 7 Oct 2026

VoCoT: Unleashing Visually Grounded Multi-Step Reasoning in Large Multi-Modal Models

arXiv 2405.16919 ↗First posted 27 May 2024cs.CV cs.AI cs.CLcited by 42

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

While large multi-modal models (LMMs) have exhibited impressive capabilities across diverse tasks, their effectiveness in handling complex tasks has been limited by the prevailing single-step reasoning paradigm. To this end, this paper proposes VoCoT, a multi-step Visually grounded object-centric Chain-of-Thought reasoning framework tailored for inference with LMMs. VoCoT is characterized by two key features: (1) object-centric reasoning paths that revolve around cross-modal shared object-level information, and (2) visually grounded representation of object concepts in a multi-modal interleaved and aligned manner, which effectively bridges the modality gap within LMMs during long-term generation. To adapt LMMs in reasoning with VoCoT, we further construct an instruction-tuning dataset. By combining VoCoT with the prevalent open-source LMM architectures, we develop a VoCoT-based model, VolCano. With only 7B parameters and limited input image resolution, VolCano demonstrates excellent performance across various scenarios. In benchmarks like CLEVR and EmbSpatial, which highly require complex reasoning capabilities, VolCano outperforms SOTA models, including powerful GPT-4V. Related code, data and models are released in https://github.com/RupertLuo/VoCoT.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 60 of its 73 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

60 of 73 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models18 Feb
  2. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks4 Oct
  3. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions19 Aug
  4. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 55

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  11. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
38 more
  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  2. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  3. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  4. 2019Towards VQA Models That Can Readcited by 1,696
  5. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  6. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  7. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  8. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  9. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  10. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  11. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  13. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  14. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  15. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  16. 2016A Diagram Is Worth A Dozen Imagescited by 793
  17. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  18. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  19. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  20. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  21. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  22. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  23. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  24. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  25. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  26. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  27. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  28. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 293
  29. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  30. 2022Visual Spatial Reasoningcited by 288
  31. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 245
  32. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 235
  33. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  34. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  35. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 192
  36. 2023SVIT: Scaling up Visual Instruction Tuningcited by 161
  37. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 160
  38. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 158

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 24; 2026: 14.

42 papers.

Sort:

By area:

  1. 2024ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Explorationcited by 99
  2. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  3. 2025Grounded Reinforcement Learning for Visual Reasoningcited by 80
  4. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  5. 2025Grounded Chain-of-Thought for Multimodal Large Language Modelscited by 57
  6. 2025Ground-R1: Incentivizing Grounded Visual Reasoning via Reinforcement Learningcited by 46
  7. 2025SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoningcited by 45
  8. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulationcited by 34
  9. 2024TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokenscited by 32
  10. 2025Argus: Vision-Centric Reasoning with Grounded Chain-of-Thoughtcited by 26
  11. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 26
  12. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
30 more
  1. 2024STEP: Enhancing Video-LLMs' Compositional Reasoning by Spatio-Temporal Graph-guided Self-Trainingcited by 22
  2. 2025CMMCoT: Enhancing Complex Multi-Image Comprehension via Multi-Modal Chain-of-Thought and Memory Augmentationcited by 22
  3. 2025Mind with Eyes: from Language Reasoning to Multimodal Reasoningcited by 22
  4. 2024From the Least to the Most: Building a Plug-and-Play Visual Reasoner via Data Synthesiscited by 17
  5. 2025GThinker: Towards General Multimodal Reasoning via Cue-Guided Rethinkingcited by 14
  6. 2025OWL: Geometry-Aware Spatial Reasoning for Audio Large Language Modelscited by 9
  7. 2025AOR: Anatomical Ontology-Guided Reasoning for Medical Large Multimodal Model in Chest X-Ray Interpretationcited by 8
  8. 2025Mixture-of-Visual-Thoughts: Exploring Context-Adaptive Reasoning Mode Selection for General Visual Reasoningcited by 7
  9. 2025Let Androids Dream of Electric Sheep: A Human-Inspired Image Implication Understanding and Reasoning Frameworkcited by 5
  10. 2025Same or Not? Enhancing Visual Perception in Vision-Language Modelscited by 5
  11. 2025Understand, Think, and Answer: Advancing Visual Reasoning with Large Multimodal Modelscited by 3
  12. 2025Multi-Step Reasoning for Embodied Question Answering via Tool Augmentationcited by 3
  13. 2025GeoViS: Geospatially Rewarded Visual Search for Remote Sensing Visual Groundingcited by 3
  14. 2025No Labels, No Problem: Training Visual Reasoners with Multimodal Verifierscited by 3
  15. 2026CARE: Towards Clinical Accountability in Multi-Modal Medical Reasoning with an Evidence-Grounded Agentic Frameworkcited by 3
  16. 2025OViP: Online Vision-Language Preference Learning for VLM Hallucinationcited by 2
  17. 2026MetaphorStar: Image Metaphor Understanding and Reasoning with End-to-End Visual Reinforcement Learningcited by 2
  18. 2026Can Vision-Language Models Solve the Shell Game?cited by 2
  19. 2026Spectral-Progressive Thought Flow for Lightweight Multimodal Reasoningcited by 1
  20. 2026VistaHop: Benchmarking Long-Horizon Visual DeepSearchcited by 1
  21. 2025MIND: Multi-rationale INtegrated Discriminative Reasoning Framework for Multi-modal Large Modelscited by 0
  22. 2026RPIQ: Residual-Projected Multi-Collaboration Closed-Loop and Single Instance Quantization for Visually Impaired Assistancecited by 0
  23. 2026SkyNative: A Native Multimodal Architecture for Remote Sensing Vision-Language Understandingcited by 0
  24. 2026RoboPIN: Grounded Embodied Reasoning via Pinned Chain-of-Thoughtcited by 0
  25. 2026Faithful Grounded Visual Reasoning via Learned Proxy-Tokenscited by 0
  26. 2026From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Modelscited by 0
  27. 2026Learning to Zoom Efficiently with a Contrastive Curriculumcited by 0
  28. 2026First Things First: Teaching LLM-Based Agents to Prioritize Must-Haves before Nice-to-Havescited by 0
  29. 2026From Perception to Integration: Revisiting the Internal Dynamics of Reasoning in Vision-Language Modelscited by 0
  30. 2026MG-Thinker: Bi-Axial Self-Reflection for Multi-Image Reasoning Groundingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VGR: Visual Grounded Reasoning
  2. 2024M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought
  3. 2025LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMs
  4. 2025RBench-V: A Primary Assessment for Visual Reasoning Models with Multi-modal Outputs
  5. 2024Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
  6. 2025VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoning
  7. 2025VOILA: Evaluation of MLLMs For Perceptual Understanding and Analogical Reasoning
  8. 2025SceneCOT: Eliciting Grounded Chain-of-Thought Reasoning in 3D Scenes