mlsanityData to 14 Oct 2026

T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Mixed Large Language Model Signals for Science Question Answering

arXiv 2305.03453 ↗First posted 5 May 2023cs.CLcited by 60

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Large Language Models (LLMs) have recently demonstrated exceptional performance in various Natural Language Processing (NLP) tasks. They have also shown the ability to perform chain-of-thought (CoT) reasoning to solve complex problems. Recent studies have explored CoT reasoning in complex multimodal scenarios, such as the science question answering task, by fine-tuning multimodal models with high-quality human-annotated CoT rationales. However, collecting high-quality COT rationales is usually time-consuming and costly. Besides, the annotated rationales are hardly accurate due to the external essential information missed. To address these issues, we propose a novel method termed T-SciQ that aims at teaching science question answering with LLM signals. The T-SciQ approach generates high-quality CoT rationales as teaching signals and is advanced to train much smaller models to perform CoT reasoning in complex modalities. Additionally, we introduce a novel data mixing strategy to produce more effective teaching data samples for simple and complex science question answer problems. Extensive experimental results show that our T-SciQ method achieves a new state-of-the-art performance on the ScienceQA benchmark, with an accuracy of 96.18%. Moreover, our approach outperforms the most powerful fine-tuned baseline by 4.5%. The code is publicly available at https://github.com/T-SciQ/T-SciQ.

Key facts. Cited by 60 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 51 of its 68 references that are in the map. First posted on arXiv 5 May 2023.

Builds on

51 of 68 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Multimodal Chain-of-Thought Reasoning in Language Models2 Feb
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep

Other references · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2023GPT-4 Technical Reportcited by 18,396
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  9. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  10. 2023Visual Instruction Tuningcited by 7,869
  11. 2020End-to-End Object Detection with Transformerscited by 7,414
  12. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
37 more
  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  2. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  3. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  4. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  5. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  6. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  7. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  8. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  9. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  10. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  11. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  12. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  13. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  14. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  15. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 688
  16. 2022Large Language Models Can Self-Improvecited by 664
  17. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  18. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  19. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 558
  20. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 476
  21. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 388
  22. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356
  23. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 350
  24. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 339
  25. 2022Large Language Models Are Reasoning Teacherscited by 329
  26. 2022Teaching Small Language Models to Reasoncited by 284
  27. 2018Bilinear Attention Networkscited by 280
  28. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 267
  29. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 251
  30. 2021Explaining Answers with Entailment Treescited by 152
  31. 2018Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answeringcited by 114
  32. 2022Rationale-Augmented Ensembles in Language Modelscited by 111
  33. 2022Iteratively Prompt Pre-trained Language Models for Chain of Thoughtcited by 75
  34. 2018WorldTree: A Corpus of Explanation Graphs for Elementary Science Questions supporting Multi-Hop Inferencecited by 60
  35. 2021Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translationcited by 44
  36. 2023ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extractioncited by 38
  37. 2022On Vision Features in Multimodal Machine Translationcited by 31

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 21; 2025: 23; 2026: 10.

60 papers.

Sort:

By area:

  1. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  2. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 182
  3. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 164
  4. 2024Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Modelscited by 133
  5. 2024M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thoughtcited by 121
  6. 2024Plan-on-Graph: Self-Correcting Adaptive Planning of Large Language Model on Knowledge Graphscited by 78
  7. 2024KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoningcited by 60
  8. 2024Interleaved-Modal Chain-of-Thoughtcited by 51
  9. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learningcited by 50
  10. 2025Ground-R1: Incentivizing Grounded Visual Reasoning via Reinforcement Learningcited by 49
  11. 2025Evaluating Scoring Bias in LLM-as-a-Judgecited by 43
  12. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 41
38 more
  1. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  2. 2025Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thoughtcited by 35
  3. 2024Cantor: Inspiring Multimodal Chain-of-Thought of MLLMcited by 34
  4. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  5. 2024DesignQA: A Multimodal Benchmark for Evaluating Large Language Models' Understanding of Engineering Documentationcited by 32
  6. 2024Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrievalcited by 32
  7. 2024Data Augmentation using Large Language Models: Data Perspectives, Learning Paradigms and Challengescited by 30
  8. 2023Thinking Like an Expert:Multimodal Hypergraph-of-Thought (HoT) Reasoning to boost Foundation Modalscited by 24
  9. 2025Can Multimodal Large Language Models Understand Spatial Relations?cited by 20
  10. 2025MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answeringcited by 19
  11. 2024Describe-then-Reason: Improving Multimodal Mathematical Reasoning through Visual Comprehension Trainingcited by 17
  12. 2025Towards LLM Agents for Earth Observationcited by 17
  13. 2025Chiron-o1: Igniting Multimodal Large Language Models towards Generalizable Medical Reasoning via Mentor-Intern Collaborative Searchcited by 16
  14. 2023Enhancing Human-like Multi-Modal Reasoning: A New Challenging Dataset and Comprehensive Frameworkcited by 11
  15. 2024A Picture Is Worth a Graph: A Blueprint Debate Paradigm for Multimodal Reasoningcited by 11
  16. 2024Multimodal Commonsense Knowledge Distillation for Visual Question Answeringcited by 11
  17. 2024ML-Mamba: Efficient Multi-Modal Large Language Model Utilizing Mamba-2cited by 10
  18. 2024A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Futurecited by 9
  19. 2025Citrus-V: Advancing Medical Foundation Models with Unified Medical Image Grounding for Clinical Reasoningcited by 8
  20. 2024SAIL: Sample-Centric In-Context Learning for Document Information Extractioncited by 6
  21. 2025CoFFT: Chain of Foresight-Focus Thought for Visual Language Modelscited by 6
  22. 2026OmniOPSD: Rationale-Privileged On-Policy Self-Distillation for Affective Computingcited by 6
  23. 2023Efficient End-to-End Visual Document Understanding with Rationale Distillationcited by 5
  24. 2024Benchmarking VLMs' Reasoning About Persuasive Atypical Imagescited by 5
  25. 2025Multimodal Mathematical Reasoning with Diverse Solving Perspectivecited by 4
  26. 2024Survey of Natural Language Processing for Education: Taxonomy, Systematic Review, and Future Trendscited by 3
  27. 2025Towards Multimodal Metaphor Understanding: A Chinese Dataset and Model for Metaphor Mapping Identificationcited by 3
  28. 2025Stepwise Decomposition and Dual-stream Focus: A Novel Approach for Training-free Camouflaged Object Segmentationcited by 3
  29. 2026Can Large Language Models Self-Correct in Medical Question Answering? An Exploratory Studycited by 3
  30. 2026Reliable Chain-of-Thought via Prefix Consistencycited by 3
  31. 2026Code-on-Graph: Iterative Programmatic Reasoning via Large Language Models on Knowledge Graphscited by 3
  32. 2023A Survey on Interpretable Cross-modal Reasoningcited by 2
  33. 2025Privacy-protected Retrieval-Augmented Generation for Knowledge Graph Question Answeringcited by 2
  34. 2025Static Sandboxes Are Inadequate: Modeling Societal Complexity Requires Open-Ended Co-Evolution in LLM-Based Multi-Agent Simulationscited by 2
  35. 2025Doc-CoB: Enhancing Document Understanding with Visual Chain-of-Boxes Reasoningcited by 1
  36. 2025Deep Hidden Cognition Facilitates Reliable Chain-of-Thought Reasoningcited by 1
  37. 2025Guided Navigation in Knowledge-Dense Environments: Structured Semantic Exploration with Guidance Graphscited by 1
  38. 2025Bolster Hallucination Detection via Prompt-Guided Data Augmentationcited by 1

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Sci-CoT: Leveraging Large Language Models for Enhanced Knowledge Distillation in Small Models for Scientific QA
  2. 2025Cross Domain Evaluation of Multimodal Chain-of-Thought Reasoning of different datasets into the Amazon CoT Framework
  3. 2024SceMQA: A Scientific College Entrance Level Multimodal Question Answering Benchmark
  4. 2023Towards a Unified Multimodal Reasoning Framework
  5. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
  6. 2023Knowledge-Driven CoT: Exploring Faithful Reasoning in LLMs for Knowledge-intensive Question Answering
  7. 2026SciVQR: A Multidisciplinary Multimodal Benchmark for Advanced Scientific Reasoning Evaluation
  8. 2024Retrieval Meets Reasoning: Even High-school Textbook Knowledge Benefits Multimodal Reasoning