mlsanityData to 7 Oct 2026

KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoning

arXiv 2401.12863 ↗First posted 23 Jan 2024cs.CL cs.AIcited by 60

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated impressive performance in natural language processing tasks by leveraging chain of thought (CoT) that enables step-by-step thinking. Extending LLMs with multimodal capabilities is the recent interest, but incurs computational cost and requires substantial hardware resources. To address these challenges, we propose KAM-CoT a framework that integrates CoT reasoning, Knowledge Graphs (KGs), and multiple modalities for a comprehensive understanding of multimodal tasks. KAM-CoT adopts a two-stage training process with KG grounding to generate effective rationales and answers. By incorporating external knowledge from KGs during reasoning, the model gains a deeper contextual understanding reducing hallucinations and enhancing the quality of answers. This knowledge-augmented CoT reasoning empowers the model to handle questions requiring external context, providing more informed answers. Experimental findings show KAM-CoT outperforms the state-of-the-art methods. On the ScienceQA dataset, we achieve an average accuracy of 93.87%, surpassing GPT-3.5 (75.17%) by 18% and GPT-4 (83.99%) by 10%. Remarkably, KAM-CoT achieves these results with only 280M trainable parameters at a time, demonstrating its cost-efficiency and effectiveness.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 34 of its 44 references that are in the map. First posted on arXiv 23 Jan 2024.

Builds on

34 of 44 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Mixed Large Language Model Signals for Science Question Answering5 May
  2. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model28 Apr
  3. 2023Multimodal Chain-of-Thought Reasoning in Language Models2 Feb
  4. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep

Other references · 30

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,414
  9. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  10. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  11. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  12. 2018Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learningcited by 1,099
18 more
  1. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  2. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  3. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  4. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  5. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  6. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  7. 2021QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answeringcited by 386
  8. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  9. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  10. 2019KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoningcited by 216
  11. 2022GreaseLM: Graph REASoning Enhanced Language Models for Question Answeringcited by 170
  12. 2020KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQAcited by 124
  13. 2020Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answeringcited by 122
  14. 2019Relational Graph Attention Networkscited by 81
  15. 2021Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translationcited by 44
  16. 2022On Vision Features in Multimodal Machine Translationcited by 31
  17. 2023SCITUNE: Aligning Large Language Models with Human-Curated Scientific Multimodal Instructionscited by 19
  18. 2020Neural Machine Translation: Challenges, Progress and Futurecited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 25; 2026: 19.

60 papers.

Sort:

By area:

  1. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 398
  2. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  3. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  4. 2024Large Language Models and Games: A Survey and Roadmapcited by 91
  5. 2025Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodologycited by 67
  6. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  7. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  8. 2024Beyond Chain-of-Thought: A Survey of Chain-of-X Paradigms for LLMscited by 48
  9. 2026Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perceptioncited by 47
  10. 2024Multimodal Reasoning with Multimodal Knowledge Graphcited by 43
  11. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  12. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
38 more
  1. 2025Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thoughtcited by 34
  2. 2024CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Modelscited by 29
  3. 2024CoReS: Orchestrating the Dance of Reasoning and Segmentationcited by 25
  4. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  5. 2024The Evolution of Multimodal Model Architecturescited by 21
  6. 2025LayerCraft: Enhancing Text-to-Image Generation with CoT Reasoning and Layered Object Integrationcited by 19
  7. 2025MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answeringcited by 18
  8. 2025Graph-based Approaches and Functionalities in Retrieval-Augmented Generation: A Comprehensive Surveycited by 18
  9. 2025Debate-to-Detect: Reformulating Misinformation Detection as a Real-World Debate with Large Language Modelscited by 12
  10. 2025Reinforced Reasoning for Embodied Planningcited by 12
  11. 2024A Picture Is Worth a Graph: A Blueprint Debate Paradigm for Multimodal Reasoningcited by 11
  12. 2024Rationale-guided Prompting for Knowledge-based Visual Question Answeringcited by 9
  13. 2025Evolving Roles of LLMs in Scientific Innovation: Assistant, Collaborator, Scientist, and Evaluatorcited by 9
  14. 2024Enhancing Large Language Models with Pseudo- and Multisource- Knowledge Graphs for Open-ended Question Answeringcited by 8
  15. 2025CoT-RVS: Zero-Shot Chain-of-Thought Reasoning Segmentation for Videoscited by 8
  16. 2025Think Before You Segment: High-Quality Reasoning Segmentation with GPT Chain of Thoughtscited by 7
  17. 2026Omni-R1: Towards the Unified Generative Paradigm for Multimodal Reasoningcited by 7
  18. 2025BottleHumor: Self-Informed Humor Explanation using the Information Bottleneck Principlecited by 6
  19. 2025LLaVA-RadZ: Can Multimodal Large Language Models Effectively Tackle Zero-shot Radiology Recognition?cited by 6
  20. 2025ChartPoint: Guiding MLLMs with Grounding Reflection for Chart Reasoningcited by 6
  21. 2024Can Agents Spontaneously Form a Society? Introducing a Novel Architecture for Generative Multi-Agents to Elicit Social Emergencecited by 5
  22. 2024Exploring Prompt Engineering: A Systematic Review with SWOT Analysiscited by 5
  23. 2025Integrating Chain-of-Thought for Multimodal Alignment: A Study on 3D Vision-Language Learningcited by 5
  24. 2025RealSR-R1: Reinforcement Learning for Real-World Image Super-Resolution with Vision-Language Chain-of-Thoughtcited by 5
  25. 2025EcoAlign: An Economically Rational Framework for Efficient LVLM Alignmentcited by 5
  26. 2026PaLMR: Towards Faithful Visual Reasoning via Multimodal Process Alignmentcited by 5
  27. 2026Visual Enhanced Depth Scaling for Multimodal Latent Reasoningcited by 3
  28. 2024Internalizing ASR with Implicit Chain of Thought for Efficient Speech-to-Speech Conversational LLMcited by 2
  29. 2026CoT-Seg: Rethinking Segmentation with Chain-of-Thought Reasoning and Self-Correctioncited by 2
  30. 2026Knowledge Injection Exists in MoE? Exploring Expert-Aware Contrast Decoding in MoE for Mitigating LLMs'Hallucinationscited by 2
  31. 2025COSINT-Agent: A Knowledge-Driven Multimodal Agent for Chinese Open Source Intelligencecited by 1
  32. 2025QG-CoC: Question-Guided Chain-of-Captions for Large Multimodal Modelscited by 1
  33. 2026Learning to Focus and Precise Cropping: A Reinforcement Learning Framework with Information Gaps and Grounding Loss for MLLMscited by 1
  34. 2026Spectral-Progressive Thought Flow for Lightweight Multimodal Reasoningcited by 1
  35. 2024How Vision-Language Tasks Benefit from Large Pre-trained Models: A Surveycited by 0
  36. 2025Rationale-Enhanced Decoding for Multi-modal Chain-of-Thoughtcited by 0
  37. 2025Augmented Vision-Language Models: A Systematic Reviewcited by 0
  38. 2026Parallel In-context Learning for Large Vision Language Modelscited by 0

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
  2. 2025Cross Domain Evaluation of Multimodal Chain-of-Thought Reasoning of different datasets into the Amazon CoT Framework
  3. 2025From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Models
  4. 2023Knowledge-Driven CoT: Exploring Faithful Reasoning in LLMs for Knowledge-intensive Question Answering
  5. 2023Towards a Unified Multimodal Reasoning Framework
  6. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning
  7. 2023Sci-CoT: Leveraging Large Language Models for Enhanced Knowledge Distillation in Small Models for Scientific QA
  8. 2025CoT Vectors: Transferring and Probing the Reasoning Mechanisms of LLMs