mlsanityData to 7 Oct 2026

Chain-of-Spot: Interactive Reasoning Improves Large Vision-Language Models

arXiv 2403.12966 ↗First posted 19 Mar 2024cs.CVcited by 65

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

In the realm of vision-language understanding, the proficiency of models in interpreting and reasoning over visual content has become a cornerstone for numerous applications. However, it is challenging for the visual encoder in Large Vision-Language Models (LVLMs) to extract useful features tailored to questions that aid the language model's response. Furthermore, a common practice among existing LVLMs is to utilize lower-resolution images, which restricts the ability for visual recognition. Our work introduces the Chain-of-Spot (CoS) method, which we describe as Interactive Reasoning, a novel approach that enhances feature extraction by focusing on key regions of interest (ROI) within the image, corresponding to the posed questions or instructions. This technique allows LVLMs to access more detailed visual information without altering the original image resolution, thereby offering multi-granularity image features. By integrating Chain-of-Spot with instruct-following LLaVA-1.5 models, the process of image reasoning consistently improves performance across a wide range of multimodal datasets and benchmarks without bells and whistles and achieves new state-of-the-art results. Our empirical findings demonstrate a significant improvement in LVLMs' ability to understand and reason about visual content, paving the way for more sophisticated visual instruction-following applications. Code and models are available at https://github.com/dongyh20/Chain-of-Spot

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 41 of its 47 references that are in the map. First posted on arXiv 19 Mar 2024.

Builds on

41 of 47 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep

Other references · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  10. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  11. 2023Visual Instruction Tuningcited by 7,792
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
27 more
  1. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  4. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  7. 2019Towards VQA Models That Can Readcited by 1,696
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  9. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,417
  10. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,275
  11. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  12. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  13. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  14. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  15. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  16. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  17. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  18. 2023Instruction Tuning with GPT-4cited by 676
  19. 2019Panoptic Feature Pyramid Networkscited by 644
  20. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  21. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  22. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 316
  23. 2020Glance and Focus: a Dynamic Approach to Reducing Spatial Redundancy in Image Classificationcited by 99
  24. 2021Adaptive Focus for Efficient Video Recognitioncited by 76
  25. 2021Dynamic Resolution Networkcited by 31
  26. 2022Dynamic Spatial Sparsification for Efficient Vision Transformers and Convolutional Neural Networkscited by 23
  27. 2024DualFocus: Integrating Macro and Micro Perspectives in Multi-modal Large Language Modelscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 37; 2026: 21.

65 papers.

Sort:

By area:

  1. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 398
  2. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  3. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  4. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157
  5. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 120
  6. 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Modelscited by 112
  7. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  8. 2025EgoLife: Towards Egocentric Life Assistantcited by 86
  9. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 79
  10. 2026Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillationcited by 71
  11. 2025VGR: Visual Grounded Reasoningcited by 70
  12. 2025Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoningcited by 67
38 more
  1. 2025Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodologycited by 67
  2. 2024Visual Agents as Fast and Slow Thinkerscited by 53
  3. 2026Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perceptioncited by 47
  4. 2025Ground-R1: Incentivizing Grounded Visual Reasoning via Reinforcement Learningcited by 46
  5. 2025MINT-CoT: Enabling Interleaved Visual Tokens in Mathematical Chain-of-Thought Reasoningcited by 45
  6. 2025ACTIVE-o3: Empowering MLLMs with Active Perception via Pure Reinforcement Learningcited by 41
  7. 2025RealUnify: Do Unified Models Truly Benefit from Unification? A Comprehensive Benchmarkcited by 33
  8. 2025Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoningcited by 28
  9. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
  10. 2025v1: Learning to Point Visual Tokens for Multimodal Grounded Reasoningcited by 24
  11. 2025Towards Video Thinking Test: A Holistic Benchmark for Advanced Video Reasoning and Understandingcited by 24
  12. 2024Efficient Inference of Vision Instruction-Following Models with Elastic Cachecited by 23
  13. 2024Token-level Correlation-guided Compression for Efficient Multimodal Document Understandingcited by 22
  14. 2025ShotBench: Expert-Level Cinematic Understanding in Vision-Language Modelscited by 22
  15. 2025Scaling Agentic Reinforcement Learning for Tool-Integrated Reasoning in VLMscited by 16
  16. 2025GThinker: Towards General Multimodal Reasoning via Cue-Guided Rethinkingcited by 14
  17. 2025Thinking in 360deg: Humanoid Visual Search in the Wildcited by 14
  18. 2024VISCO: Benchmarking Fine-Grained Critique and Correction Towards Self-Improvement in Visual Reasoningcited by 13
  19. 2025High-Resolution Visual Reasoning via Multi-Turn Grounding-Based Reinforcement Learningcited by 13
  20. 2025PixelCraft: A Multi-Agent System for High-Fidelity Visual Reasoning on Structured Imagescited by 11
  21. 2025GUI-ARP: Enhancing Grounding with Adaptive Region Perception for GUI Agentscited by 10
  22. 2025WaymoQA: A Multi-View Visual Question Answering Dataset for Safety-Critical Reasoning in Autonomous Drivingcited by 9
  23. 2025Video-CoM: Interactive Video Reasoning via Chain of Manipulationscited by 9
  24. 20253EED: Ground Everything Everywhere in 3Dcited by 8
  25. 2025TARAC: Mitigating Hallucination in LVLMs via Temporal Attention Real-time Accumulative Connectioncited by 7
  26. 2025Evidence-Grounded Trustworthy Multimodal Reasoning and Evaluation Benchmark in Complex Urban Scenescited by 7
  27. 2026SketchVL: Policy Optimization via Fine-Grained Credit Assignment for Chart Understanding and Morecited by 7
  28. 2025Unveiling Chain of Step Reasoning for Vision-Language Models with Fine-grained Rewardscited by 6
  29. 2026M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understandingcited by 6
  30. 2025LIRA: Inferring Segmentation in Large Multi-modal Models with Local Interleaved Region Assistancecited by 5
  31. 2025VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoningcited by 5
  32. 2026Adaptive Time Series Reasoning via Segment Selectioncited by 3
  33. 2026HART: High-Resolution Annotation-Free Reasoning Technique through a Closed-loop Frameworkcited by 3
  34. 2026Perceive-to-Reason: Decoupling Perception and Reasoning for Fine-Grained Visual Reasoningcited by 3
  35. 2026Insight-V++: Towards Advanced Long-Chain Visual Reasoning with Multimodal Large Language Modelscited by 2
  36. 2026Thinking With Tools, Not With Pixels: Tool Calls as Text Scaffolds for Visual Reasoningcited by 2
  37. 2025GroundSight: Augmenting Vision-Language Models with Grounding Information and De-hallucinationcited by 1
  38. 2025The Impact of Image Resolution on Biomedical Multimodal Large Language Modelscited by 1

50 most cited of 65 All 65 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Weaving Context Across Images: Improving Vision-Language Models through Focus-Centric Visual Chains
  2. 2024Enhancing Advanced Visual Reasoning Ability of Large Language Models
  3. 2025Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens
  4. 2024Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning
  5. 2024From the Least to the Most: Building a Plug-and-Play Visual Reasoner via Data Synthesis
  6. 2024Zero-Shot Visual Reasoning by Vision-Language Models: Benchmarking and Analysis
  7. 2026VG-CoT: Towards Trustworthy Visual Reasoning via Grounded Chain-of-Thought
  8. 2024Advancing Large Multi-modal Models with Explicit Chain-of-Reasoning and Visual Question Generation