mlsanityData to 7 Oct 2026

Retrieval-Based Interleaved Visual Chain-of-Thought in Real-World Driving Scenarios

arXiv 2501.04671 ↗First posted 8 Jan 2025cs.CV cs.AIcited by 16

On the map: LLM reasoning → Multimodal reasoning

also in 2 more
Abstract (arXiv)

While chain-of-thought (CoT) prompting improves reasoning in large language models, its effectiveness in vision-language models (VLMs) remains limited due to over-reliance on textual cues and memorized knowledge. To investigate the visual reasoning capabilities of VLMs in complex real-world scenarios, we introduce DrivingVQA, a visual question answering dataset derived from driving theory exams, which contains 3,931 multiple-choice problems with expert-written explanations and grounded entities relevant to the reasoning process. Leveraging this dataset, we propose RIV-CoT, a Retrieval-Based Interleaved Visual Chain-of-Thought method that enables VLMs to reason using visual crops corresponding to these relevant entities. Our experiments demonstrate that RIV-CoT improves answer accuracy by 3.1% and reasoning accuracy by 4.6% over vanilla CoT prompting. Furthermore, we demonstrate that our method effectively scales to the larger A-OKVQA reasoning dataset by leveraging automatically generated pseudo-labels, outperforming CoT prompting.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 16 of its 59 references that are in the map. First posted on arXiv 8 Jan 2025.

Builds on

16 of 59 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning25 Mar
  2. 2024PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patterns20 Mar
  3. 2024CogCoM: A Visual Language Model with Chain-of-Manipulations Reasoning6 Feb

Other references · 13

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  4. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  5. 2024A Survey on LLM-as-a-Judgecited by 1,384
  6. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  7. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  8. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
  9. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 226
  10. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 146
  11. 2024Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Modelscited by 56
  12. 2024List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMscited by 32
1 more
  1. 2024Response Wide Shut: Surprising Observations in Basic Vision Language Model Capabilitiescited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 9.

16 papers.

Sort:

By area:

  1. 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tailcited by 136
  2. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 105
  3. 2026Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodimentscited by 42
  4. 2025CodePlot-CoT: Mathematical Visual Reasoning by Thinking with Code-Driven Imagescited by 22
  5. 2025Reasoning in Computer Vision: Taxonomy, Models, Tasks, and Methodologiescited by 17
  6. 2025Latent Chain-of-Thought World Modeling for End-to-End Drivingcited by 14
  7. 2025Chain-of-Thought for Autonomous Driving: A Comprehensive Survey and Future Prospectscited by 12
  8. 2025OmniDrive-R1: Reinforcement-driven Interleaved Multi-modal Chain-of-Thought for Trustworthy Vision-Language Autonomous Drivingcited by 9
  9. 2026ThinkDrive: Chain-of-Thought Guided Progressive Reinforcement Learning Fine-Tuning for Autonomous Drivingcited by 3
  10. 2026RailVQA: A Benchmark and Framework for Efficient Interpretable Visual Cognition in Automatic Train Operationcited by 1
  11. 2026Hard Vision, Easy Vision: What GPT-6 Astra Reveals Across Computer Visioncited by 1
  12. 2026From Steering to Pedalling: Do Autonomous Driving VLMs Generalize to Cyclist-Assistive Spatial Perception and Planning?cited by 0
4 more
  1. 2026CaVe-VLM-CoT: An Interpretable Vision-Language Model Frameworkcited by 0
  2. 2026MentalThink: Shaping Thoughts in Mental SVG Worldcited by 0
  3. 2026Aphanta: Diagnosing Task-Aligned Image-Edited Intermediates for Multimodal Reasoningcited by 0
  4. 2026Beyond Textual Chain-of-Thought: A Survey on Action-Grounded Reasoning in Autonomous Drivingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generation
  2. 2026TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding
  3. 2024II-MMR: Identifying and Improving Multi-modal Multi-hop Reasoning in Visual Question Answering
  4. 2023Towards a Unified Multimodal Reasoning Framework
  5. 2025VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoning
  6. 2024Advancing Large Multi-modal Models with Explicit Chain-of-Reasoning and Visual Question Generation
  7. 2023Investigating Prompting Techniques for Zero- and Few-Shot Visual Question Answering
  8. 2026Attention-guided Fine-tuning of Multimodal Large Language Models Improves Chain-of-Thought Reasoning