# SceMQA: A Scientific College Entrance Level Multimodal Question Answering Benchmark

arXiv: https://arxiv.org/abs/2402.05138 · HTML: https://mlsanity.com/paper/2402.05138/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 51 references that are in the map.
- First posted on arXiv 6 Feb 2024.

## Builds on (most cited first, 30 of 32)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [Are NLP Models really able to Solve Simple Math Word Problems?](https://mlsanity.com/paper/2103.07191/) (2021)
- [Large Language Model based Multi-Agents: A Survey of Progress and Challenges](https://mlsanity.com/paper/2402.01680/) (2024)
- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning](https://mlsanity.com/paper/2309.05653/) (2023)
- [Large Language Models Are Reasoning Teachers](https://mlsanity.com/paper/2212.10071/) (2022)
- [LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding](https://mlsanity.com/paper/2306.17107/) (2023)
- [SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models](https://mlsanity.com/paper/2307.10635/) (2023)
- [Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2311.13165/) (2023)
- [Evaluating the Performance of Large Language Models on GAOKAO Benchmark](https://mlsanity.com/paper/2305.12474/) (2023)
- [Orca 2: Teaching Small Language Models How to Reason](https://mlsanity.com/paper/2311.11045/) (2023)
- [VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use](https://mlsanity.com/paper/2308.06595/) (2023)
- [Let GPT be a Math Tutor: Teaching Math Word Problem Solvers with Customized Exercise Generation](https://mlsanity.com/paper/2305.14386/) (2023)
- [MinT: Boosting Generalization in Mathematical Reasoning via Multi-View Fine-Tuning](https://mlsanity.com/paper/2307.07951/) (2023)

## Built on by (most cited first, 30 of 35)

- [Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models](https://mlsanity.com/paper/2503.06749/) (2025)
- [Vision language models are blind: Failing to translate detailed visual features into words](https://mlsanity.com/paper/2407.06581/) (2024)
- [A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery](https://mlsanity.com/paper/2406.10833/) (2024)
- [A Survey on Benchmarks of Multimodal Large Language Models](https://mlsanity.com/paper/2408.08632/) (2024)
- [A Survey on Evaluation of Multimodal Large Language Models](https://mlsanity.com/paper/2408.15769/) (2024)
- [Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoning](https://mlsanity.com/paper/2502.02871/) (2025)
- [Probing the limitations of multimodal language models for chemistry and materials research](https://mlsanity.com/paper/2411.16955/) (2024)
- [Defending Jailbreak Prompts via In-Context Adversarial Game](https://mlsanity.com/paper/2402.13148/) (2024)
- [Qwen Look Again: Guiding Vision-Language Reasoning Models to Re-attention Visual Information](https://mlsanity.com/paper/2505.23558/) (2025)
- [A Survey on Multimodal Benchmarks: In the Era of Large AI Models](https://mlsanity.com/paper/2409.18142/) (2024)
- [VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videos](https://mlsanity.com/paper/2506.10857/) (2025)
- [SciVerse: Unveiling the Knowledge Comprehension and Visual Reasoning of LMMs on Multi-modal Scientific Problems](https://mlsanity.com/paper/2503.10627/) (2025)
- [ChemEval: A Comprehensive Multi-Level Chemical Evaluation for Large Language Models](https://mlsanity.com/paper/2409.13989/) (2024)
- [Towards LLM Agents for Earth Observation](https://mlsanity.com/paper/2504.12110/) (2025)
- [HSSBench: Benchmarking Humanities and Social Sciences Ability for Multimodal Large Language Models](https://mlsanity.com/paper/2506.03922/) (2025)
- [Dissecting Logical Reasoning in LLMs: A Fine-Grained Evaluation and Supervision Study](https://mlsanity.com/paper/2506.04810/) (2025)
- [EEE-Bench: A Comprehensive Multimodal Electrical And Electronics Engineering Benchmark](https://mlsanity.com/paper/2411.01492/) (2024)
- [VISCO: Benchmarking Fine-Grained Critique and Correction Towards Self-Improvement in Visual Reasoning](https://mlsanity.com/paper/2412.02172/) (2024)
- [Observe-R1: Unlocking Reasoning Abilities of MLLMs with Dynamic Progressive Reinforcement Learning](https://mlsanity.com/paper/2505.12432/) (2025)
- [A Causality-aware Paradigm for Evaluating Creativity of Multimodal Large Language Models](https://mlsanity.com/paper/2501.15147/) (2025)
- [MMSciBench: Benchmarking Language Models on Chinese Multimodal Scientific Problems](https://mlsanity.com/paper/2503.01891/) (2025)
- [MSEarth: A Multimodal Benchmark for Earth Science Phenomenon Discovery with MLLMs](https://mlsanity.com/paper/2505.20740/) (2025)
- [QCBench: Evaluating Large Language Models on Domain-Specific Quantitative Chemistry](https://mlsanity.com/paper/2508.01670/) (2025)
- [Unveiling Chain of Step Reasoning for Vision-Language Models with Fine-grained Rewards](https://mlsanity.com/paper/2509.19003/) (2025)
- [Self-Improvement in Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2510.02665/) (2025)
- [Beyond Single-Value Metrics: Evaluating and Enhancing LLM Unlearning with Cognitive Diagnosis](https://mlsanity.com/paper/2502.13996/) (2025)
- [RxnBench: A Multimodal Benchmark for Evaluating Large Language Models on Chemical Reaction Understanding from Scientific Literature](https://mlsanity.com/paper/2512.23565/) (2025)
- [GRAFT: GRaPH and Table Reasoning for Textual Alignment -- A Benchmark for Structured Instruction Following and Visual Reasoning](https://mlsanity.com/paper/2508.15690/) (2025)
- [ALDEN: Reinforcement Learning for Active Navigation and Evidence Gathering in Long Documents](https://mlsanity.com/paper/2510.25668/) (2025)
- [Argus Inspection: Do Multimodal Large Language Models Possess the Eye of Panoptes?](https://mlsanity.com/paper/2506.14805/) (2025)

## Abstract (arXiv)

The paper introduces SceMQA, a novel benchmark for scientific multimodal question answering at the college entrance level. It addresses a critical educational phase often overlooked in existing benchmarks, spanning high school to pre-college levels. SceMQA focuses on core science subjects including Mathematics, Physics, Chemistry, and Biology. It features a blend of multiple-choice and free-response formats, ensuring a comprehensive evaluation of AI models' abilities. Additionally, our benchmark provides specific knowledge points for each problem and detailed explanations for each answer. SceMQA also uniquely presents problems with identical contexts but varied questions to facilitate a more thorough and accurate assessment of reasoning capabilities. In the experiment, we evaluate both open-source and close-source state-of-the-art Multimodal Large Language Models (MLLMs), across various experimental settings. The results show that further research and development are needed in developing more capable MLLM, as highlighted by only 50% to 60% accuracy achieved by the strongest models. Our benchmark and analysis will be available at https://scemqa.github.io/