mlsanityData to 7 Oct 2026

Retrieval-augmented Multi-modal Chain-of-Thoughts Reasoning for Large Language Models

arXiv 2312.01714 ↗First posted 4 Dec 2023cs.CLcited by 18

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

The advancement of Large Language Models (LLMs) has brought substantial attention to the Chain of Thought (CoT) approach, primarily due to its ability to enhance the capability of LLMs on complex reasoning tasks. Moreover, the significance of CoT approaches extends to the application of LLMs for multi-modal tasks. However, the selection of optimal CoT demonstration examples in multi-modal reasoning remains less explored for LLMs due to the inherent complexity of multi-modal examples. In this paper, we introduce a novel approach that addresses this challenge by using retrieval mechanisms to dynamically and automatically select demonstration examples based on cross-modal and intra-modal similarities. Furthermore, we employ a Stratified Sampling method of categorising demonstration examples into groups based on their types and then retrieving examples from different groups respectively to promote the diversity of demonstration examples. Through a series of experiments on two popular benchmark datasets: ScienceQA and MathVista, we demonstrate that our approach significantly improves the performance of GPT-4 by 6% on ScienceQA and 12.9% on MathVista, and enhances the performance of GPT-4V on two datasets by 2.7%, substantially improving the performance of the most advanced LLMs and LMMs for complex multi-modal reasoning tasks.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 56 of its 73 references that are in the map. First posted on arXiv 4 Dec 2023.

Builds on

56 of 73 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  2. 2023Multimodal Chain-of-Thought Reasoning in Language Models2 Feb
  3. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep

Other references · 53

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  8. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  3. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  4. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  5. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  6. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  7. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  8. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  9. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  10. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  11. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  12. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  13. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  14. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  15. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  16. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  17. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  18. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  19. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  20. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  21. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 317
  22. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  23. 2022Visual Prompting via Image Inpaintingcited by 248
  24. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 221
  25. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 216
  26. 2023SegGPT: Segmenting Everything In Contextcited by 193
  27. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
  28. 2023What Makes Good Examples for Visual In-Context Learning?cited by 150
  29. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memorycited by 122
  30. 2023In-Context Learning Unlocked for Diffusion Modelscited by 103
  31. 2023PRODIGY: Enabling In-context Learning Over Graphscited by 97
  32. 2022FiD-Light: Efficient and Effective Retrieval-Augmented Text Generationcited by 63
  33. 2023Exploring Effective Factors for Improving Visual In-Context Learningcited by 43
  34. 2023Tab-CoT: Zero-shot Tabular Chain of Thoughtcited by 42
  35. 2023Multimodal Procedural Planning via Dual Text-Image Promptingcited by 40
  36. 2023A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysiscited by 37
  37. 2023GPT4Video: A Unified Multimodal Large Language Model for lnstruction-Followed Understanding and Safety-Aware Generationcited by 36
  38. 2023A Comprehensive Evaluation of GPT-4V on Knowledge-Intensive Visual Question Answeringcited by 23

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 9; 2025: 4; 2026: 4.

18 papers.

Sort:

By area:

  1. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
  2. 2024Benchmarking LLMs via Uncertainty Quantificationcited by 124
  3. 2023DrugAssist: A Large Language Model for Molecule Optimizationcited by 57
  4. 2024CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Modelscited by 29
  5. 2025Are Smarter LLMs Safer? Exploring Safety-Reasoning Trade-offs in Prompting and Fine-Tuningcited by 19
  6. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
  7. 2024Retrieval Meets Reasoning: Even High-school Textbook Knowledge Benefits Multimodal Reasoningcited by 13
  8. 2024AVG-LLaVA: An Efficient Large Multimodal Model with Adaptive Visual Granularitycited by 9
  9. 2024UniRAG: Universal Retrieval Augmentation for Large Vision Language Modelscited by 8
  10. 2024Improving Multimodal LLMs Ability In Geometry Problem Solving, Reasoning, And Multistep Scoringcited by 5
  11. 2026Unlocking the Black Box of Latent Reasoning: An Interpretability-Guided Approach to Interventioncited by 2
  12. 2025Understanding Financial Reasoning in AI: A Multimodal Benchmark and Error Learning Approachcited by 1
6 more
  1. 2025From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Modelscited by 1
  2. 2024An Overview and Discussion on Using Large Language Models for Implementation Generation of Solutions to Open-Ended Problemscited by 0
  3. 2025MIND: Multi-rationale INtegrated Discriminative Reasoning Framework for Multi-modal Large Modelscited by 0
  4. 2026SGG-Rrm 3: From Next-Token Prediction to End-to-End Unbiased Scene Graph Generationcited by 0
  5. 2026Robo-Cortex: A Self-Evolving Embodied Agent via Dual-Grain Cognitive Memory and Autonomous Knowledge Inductioncited by 0
  6. 2026Evolution of Multimodal Question Answering: From Modality-Adaptive Extraction to Unified Language Representationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance
  2. 2024M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought
  3. 2023Multi-modal Latent Space Learning for Chain-of-Thought Reasoning in Language Models
  4. 2025Chain or tree? Re-evaluating complex reasoning from the perspective of a matrix of thought
  5. 2025Reasoning Beyond Language: A Comprehensive Survey on Latent Chain-of-Thought Reasoning
  6. 2023ThoughtSource: A central hub for large language model reasoning data
  7. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
  8. 2024Multi-Step Reasoning with Large Language Models, a Survey