# MM-PhyQA: Multimodal Physics Question-Answering With Multi-Image CoT Prompting

arXiv: https://arxiv.org/abs/2404.08704 · HTML: https://mlsanity.com/paper/2404.08704/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 28 references that are in the map.
- First posted on arXiv 11 Apr 2024.

## Builds on (most cited first, 24 of 24)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)
- [Graph of Thoughts: Solving Elaborate Problems with Large Language Models](https://mlsanity.com/paper/2308.09687/) (2023)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [Multimodal Chain-of-Thought Reasoning in Language Models](https://mlsanity.com/paper/2302.00923/) (2023)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models](https://mlsanity.com/paper/2307.10635/) (2023)
- [GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning](https://mlsanity.com/paper/2105.14517/) (2021)
- [Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models](https://mlsanity.com/paper/2305.15074/) (2023)
- [A Survey on Table Question Answering: Recent Advances](https://mlsanity.com/paper/2207.05270/) (2022)

## Built on by (most cited first, 15 of 15)

- [Zebra-CoT: A Dataset for Interleaved Vision Language Reasoning](https://mlsanity.com/paper/2507.16746/) (2025)
- [MatSciBench: Benchmarking the Reasoning Ability of Large Language Models in Materials Science](https://mlsanity.com/paper/2510.12171/) (2025)
- [MM-PhyRLHF: Reinforcement Learning Framework for Multimodal Physics Question-Answering](https://mlsanity.com/paper/2404.12926/) (2024)
- [K12Vista: Exploring the Boundaries of MLLMs in K-12 Education](https://mlsanity.com/paper/2506.01676/) (2025)
- [Depression Detection and Analysis using Large Language Models on Textual and Audio-Visual Modalities](https://mlsanity.com/paper/2407.06125/) (2024)
- [Exploring Prompt Engineering: A Systematic Review with SWOT Analysis](https://mlsanity.com/paper/2410.12843/) (2024)
- [Domain Specific Benchmarks for Evaluating Multimodal Large Language Models](https://mlsanity.com/paper/2506.12958/) (2025)
- [Knowledge Graphs are all you need: Leveraging KGs in Physics Question Answering](https://mlsanity.com/paper/2412.05453/) (2024)
- [Long-context Non-factoid Question Answering in Indic Languages](https://mlsanity.com/paper/2504.13615/) (2025)
- [A Survey of Large Language Models in Discipline-specific Research: Challenges, Methods and Opportunities](https://mlsanity.com/paper/2507.08425/) (2025)
- [Steps are all you need: Rethinking STEM Education with Prompt Engineering](https://mlsanity.com/paper/2412.05023/) (2024)
- [Challenges for AI in Multimodal STEM Assessments: a Human-AI Comparison](https://mlsanity.com/paper/2507.03013/) (2025)
- [A Dialogue-Based Framework for Correcting Multimodal Errors in AI-Assisted STEM Education](https://mlsanity.com/paper/2605.04131/) (2026)
- [Su-RoBERTa: A Semi-supervised Approach to Predicting Suicide Risk through Social Media using Base Language Models](https://mlsanity.com/paper/2412.01353/) (2024)
- [FeynmanBench: Benchmarking Multimodal LLMs on Diagrammatic Physics Reasoning](https://mlsanity.com/paper/2604.03893/) (2026)

## Abstract (arXiv)

While Large Language Models (LLMs) can achieve human-level performance in various tasks, they continue to face challenges when it comes to effectively tackling multi-step physics reasoning tasks. To identify the shortcomings of existing models and facilitate further research in this area, we curated a novel dataset, MM-PhyQA, which comprises well-constructed, high schoollevel multimodal physics problems. By evaluating the performance of contemporary LLMs that are publicly available, both with and without the incorporation of multimodal elements in these problems, we aim to shed light on their capabilities. For generating answers for questions consisting of multimodal input (in this case, images and text) we employed Zero-shot prediction using GPT-4 and utilized LLaVA (LLaVA and LLaVA-1.5), the latter of which were fine-tuned on our dataset. For evaluating the performance of LLMs consisting solely of textual input, we tested the performance of the base and fine-tuned versions of the Mistral-7B and LLaMA2-7b models. We also showcased the performance of the novel Multi-Image Chain-of-Thought (MI-CoT) Prompting technique, which when used to train LLaVA-1.5 13b yielded the best results when tested on our dataset, with superior scores in most metrics and the highest accuracy of 71.65% on the test set.