# T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Mixed Large Language Model Signals for Science Question Answering

arXiv: https://arxiv.org/abs/2305.03453 · HTML: https://mlsanity.com/paper/2305.03453/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 51 of its 68 references that are in the map.
- First posted on arXiv 5 May 2023.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 30 of 51)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022)
- [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://mlsanity.com/paper/2112.00114/) (2021)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention](https://mlsanity.com/paper/2303.16199/) (2023)
- [Multimodal Chain-of-Thought Reasoning in Language Models](https://mlsanity.com/paper/2302.00923/) (2023)
- [Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies](https://mlsanity.com/paper/2101.02235/) (2021)
- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [Automatic Chain of Thought Prompting in Large Language Models](https://mlsanity.com/paper/2210.03493/) (2022)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)

## Built on by (most cited first, 30 of 60)

- [Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future](https://mlsanity.com/paper/2309.15402/) (2023)
- [Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey](https://mlsanity.com/paper/2503.12605/) (2025)
- [Compositional Chain-of-Thought Prompting for Large Multimodal Models](https://mlsanity.com/paper/2311.17076/) (2023)
- [Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Models](https://mlsanity.com/paper/2406.17294/) (2024)
- [M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought](https://mlsanity.com/paper/2405.16473/) (2024)
- [Plan-on-Graph: Self-Correcting Adaptive Planning of Large Language Model on Knowledge Graphs](https://mlsanity.com/paper/2410.23875/) (2024)
- [KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoning](https://mlsanity.com/paper/2401.12863/) (2024)
- [Interleaved-Modal Chain-of-Thought](https://mlsanity.com/paper/2411.19488/) (2024)
- [Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning](https://mlsanity.com/paper/2406.15334/) (2024)
- [Ground-R1: Incentivizing Grounded Visual Reasoning via Reinforcement Learning](https://mlsanity.com/paper/2505.20272/) (2025)
- [Evaluating Scoring Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2506.22316/) (2025)
- [A Survey on Data Synthesis and Augmentation for Large Language Models](https://mlsanity.com/paper/2410.12896/) (2024)
- [Visual Large Language Models for Generalized and Specialized Applications](https://mlsanity.com/paper/2501.02765/) (2025)
- [Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought](https://mlsanity.com/paper/2505.15510/) (2025)
- [Cantor: Inspiring Multimodal Chain-of-Thought of MLLM](https://mlsanity.com/paper/2404.16033/) (2024)
- [Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoning](https://mlsanity.com/paper/2502.02871/) (2025)
- [DesignQA: A Multimodal Benchmark for Evaluating Large Language Models' Understanding of Engineering Documentation](https://mlsanity.com/paper/2404.07917/) (2024)
- [Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2412.11077/) (2024)
- [Data Augmentation using Large Language Models: Data Perspectives, Learning Paradigms and Challenges](https://mlsanity.com/paper/2403.02990/) (2024)
- [Thinking Like an Expert:Multimodal Hypergraph-of-Thought (HoT) Reasoning to boost Foundation Modals](https://mlsanity.com/paper/2308.06207/) (2023)
- [Can Multimodal Large Language Models Understand Spatial Relations?](https://mlsanity.com/paper/2505.19015/) (2025)
- [MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answering](https://mlsanity.com/paper/2503.18491/) (2025)
- [Describe-then-Reason: Improving Multimodal Mathematical Reasoning through Visual Comprehension Training](https://mlsanity.com/paper/2404.14604/) (2024)
- [Towards LLM Agents for Earth Observation](https://mlsanity.com/paper/2504.12110/) (2025)
- [Chiron-o1: Igniting Multimodal Large Language Models towards Generalizable Medical Reasoning via Mentor-Intern Collaborative Search](https://mlsanity.com/paper/2506.16962/) (2025)
- [Enhancing Human-like Multi-Modal Reasoning: A New Challenging Dataset and Comprehensive Framework](https://mlsanity.com/paper/2307.12626/) (2023)
- [A Picture Is Worth a Graph: A Blueprint Debate Paradigm for Multimodal Reasoning](https://mlsanity.com/paper/2403.14972/) (2024)
- [Multimodal Commonsense Knowledge Distillation for Visual Question Answering](https://mlsanity.com/paper/2411.02722/) (2024)
- [ML-Mamba: Efficient Multi-Modal Large Language Model Utilizing Mamba-2](https://mlsanity.com/paper/2407.19832/) (2024)
- [A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Future](https://mlsanity.com/paper/2412.14056/) (2024)

## Abstract (arXiv)

Large Language Models (LLMs) have recently demonstrated exceptional performance in various Natural Language Processing (NLP) tasks. They have also shown the ability to perform chain-of-thought (CoT) reasoning to solve complex problems. Recent studies have explored CoT reasoning in complex multimodal scenarios, such as the science question answering task, by fine-tuning multimodal models with high-quality human-annotated CoT rationales. However, collecting high-quality COT rationales is usually time-consuming and costly. Besides, the annotated rationales are hardly accurate due to the external essential information missed. To address these issues, we propose a novel method termed T-SciQ that aims at teaching science question answering with LLM signals. The T-SciQ approach generates high-quality CoT rationales as teaching signals and is advanced to train much smaller models to perform CoT reasoning in complex modalities. Additionally, we introduce a novel data mixing strategy to produce more effective teaching data samples for simple and complex science question answer problems. Extensive experimental results show that our T-SciQ method achieves a new state-of-the-art performance on the ScienceQA benchmark, with an accuracy of 96.18%. Moreover, our approach outperforms the most powerful fine-tuned baseline by 4.5%. The code is publicly available at https://github.com/T-SciQ/T-SciQ.