mlsanityData to 14 Oct 2026

Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering

arXiv 2209.09513 ↗First posted 20 Sep 2022cs.CL cs.AI cs.CVcited by 1,919

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

When answering a question, humans utilize the information available across different modalities to synthesize a consistent and complete chain of thought (CoT). This process is normally a black box in the case of deep learning models like large-scale language models. Recently, science question benchmarks have been used to diagnose the multi-hop reasoning ability and interpretability of an AI system. However, existing datasets fail to provide annotations for the answers, or are restricted to the textual-only modality, small scales, and limited domain diversity. To this end, we present Science Question Answering (ScienceQA), a new benchmark that consists of ~21k multimodal multiple choice questions with a diverse set of science topics and annotations of their answers with corresponding lectures and explanations. We further design language models to learn to generate lectures and explanations as the chain of thought (CoT) to mimic the multi-hop reasoning process when answering ScienceQA questions. ScienceQA demonstrates the utility of CoT in language models, as CoT improves the question answering performance by 1.20% in few-shot GPT-3 and 3.99% in fine-tuned UnifiedQA. We also explore the upper bound for models to leverage explanations by feeding those in the input; we observe that it improves the few-shot performance of GPT-3 by 18.96%. Our analysis further shows that language models, similar to humans, benefit from explanations to learn from fewer data and achieve the same performance with just 40% of the data. The data and code are available at https://scienceqa.github.io.

Key facts. Cited by 1,919 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 47 of its 76 references that are in the map. First posted on arXiv 20 Sep 2022.

Builds on

47 of 76 references.

References · 47

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  11. 2015VQA: Visual Question Answeringcited by 2,506
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
35 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  2. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  3. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  4. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  5. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  6. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  7. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  8. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  9. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  10. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  11. 2016A Diagram Is Worth A Dozen Imagescited by 804
  12. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  13. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 476
  14. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 383
  15. 2021True Few-Shot Learning with Language Modelscited by 344
  16. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 340
  17. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 339
  18. 2015Visual7W: Grounded Question Answering in Imagescited by 318
  19. 2018Bilinear Attention Networkscited by 280
  20. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 267
  21. 2019QASC: A Dataset for Question Answering via Sentence Compositioncited by 252
  22. 2022Can language models learn from explanations in context?cited by 249
  23. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  24. 2021Explaining Answers with Entailment Treescited by 152
  25. 2020WT5?! Training Text-to-Text Models to Explain their Predictionscited by 148
  26. 2018Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answeringcited by 114
  27. 2015Yin and Yang: Balancing and Answering Binary Visual Questionscited by 109
  28. 2021FLEX: Unifying Evaluation for Few-Shot NLPcited by 67
  29. 2018WorldTree: A Corpus of Explanation Graphs for Elementary Science Questions supporting Multi-Hop Inferencecited by 60
  30. 2020Learning to Explain: Datasets and Models for Identifying Valid Reasoning Chains in Multihop Question-Answeringcited by 49
  31. 2018R-VQA: Learning Visual Relation Facts with Semantic Attention for Visual Question Answeringcited by 23
  32. 2022In-BoXBART: Get Instructions into Biomedical Multi-Task Learningcited by 16
  33. 2017Co-attending Free-form Regions and Detections with Multi-modal Multiplicative Feature Embedding for Visual Question Answeringcited by 13
  34. 2018Question-Guided Hybrid Convolution for Visual Question Answeringcited by 12
  35. 2016Semantic Parsing to Probabilistic Programs for Situated Question Answeringcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 9; 2023: 176; 2024: 528; 2025: 651; 2026: 555.

1,919 papers.

Sort:

By area:

  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  4. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  8. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  9. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  11. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  12. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
38 more
  1. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  2. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  3. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  4. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  5. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  6. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  7. 2023A Survey on Multimodal Large Language Modelscited by 909
  8. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  9. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  10. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  11. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  12. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  13. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  14. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  15. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  16. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  17. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 665
  18. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 642
  19. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  20. 2022Towards Reasoning in Large Language Models: A Surveycited by 596
  21. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  22. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 546
  23. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  24. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  25. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  26. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 454
  27. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  28. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 438
  29. 2023Augmented Language Models: a Surveycited by 403
  30. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 402
  31. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 391
  32. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  33. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
  34. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  35. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  36. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356
  37. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 353
  38. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 350

50 most cited of 1,919 All 1,919 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Cross Domain Evaluation of Multimodal Chain-of-Thought Reasoning of different datasets into the Amazon CoT Framework
  2. 2023Towards a Unified Multimodal Reasoning Framework
  3. 2025Enhancing Scientific Visual Question Answering through Multimodal Reasoning and Ensemble Modeling
  4. 2026M3-VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question Answering
  5. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
  6. 2023Graph Elicitation for Guiding Multi-Step Reasoning in Large Language Models
  7. 2024Direct Evaluation of Chain-of-Thought in Multi-hop Reasoning with Knowledge Graphs
  8. 2023Leveraging Structured Information for Explainable Multi-hop Question Answering and Reasoning