mlsanityData to 7 Oct 2026

SceMQA: A Scientific College Entrance Level Multimodal Question Answering Benchmark

arXiv 2402.05138 ↗First posted 6 Feb 2024cs.AI cs.CLcited by 35

Not on the map. Why?

Cited by 21 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The paper introduces SceMQA, a novel benchmark for scientific multimodal question answering at the college entrance level. It addresses a critical educational phase often overlooked in existing benchmarks, spanning high school to pre-college levels. SceMQA focuses on core science subjects including Mathematics, Physics, Chemistry, and Biology. It features a blend of multiple-choice and free-response formats, ensuring a comprehensive evaluation of AI models' abilities. Additionally, our benchmark provides specific knowledge points for each problem and detailed explanations for each answer. SceMQA also uniquely presents problems with identical contexts but varied questions to facilitate a more thorough and accurate assessment of reasoning capabilities. In the experiment, we evaluate both open-source and close-source state-of-the-art Multimodal Large Language Models (MLLMs), across various experimental settings. The results show that further research and development are needed in developing more capable MLLM, as highlighted by only 50% to 60% accuracy achieved by the strongest models. Our benchmark and analysis will be available at https://scemqa.github.io/

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 32 of its 51 references that are in the map. First posted on arXiv 6 Feb 2024.

Builds on

32 of 51 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2015VQA: Visual Question Answeringcited by 2,489
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  11. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  12. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
20 more
  1. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  2. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  3. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  4. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  5. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
  6. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  7. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  8. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  9. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  10. 2022Large Language Models Are Reasoning Teacherscited by 318
  11. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  12. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 246
  13. 2023Multimodal Large Language Models: A Surveycited by 193
  14. 2023Evaluating the Performance of Large Language Models on GAOKAO Benchmarkcited by 182
  15. 2023Orca 2: Teaching Small Language Models How to Reasoncited by 157
  16. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Usecited by 100
  17. 2023Let GPT be a Math Tutor: Teaching Math Word Problem Solvers with Customized Exercise Generationcited by 27
  18. 2023MinT: Boosting Generalization in Mathematical Reasoning via Multi-View Fine-Tuningcited by 16
  19. 2021NOAHQA: Numerical Reasoning with Interpretable Graph Question Answering Datasetcited by 14
  20. 2022Unbiased Math Word Problems Benchmark for Mitigating Solving Biascited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 22; 2026: 3.

35 papers.

Sort:

By area:

  1. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 653
  2. 2024Vision language models are blind: Failing to translate detailed visual features into wordscited by 144
  3. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  4. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  5. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  6. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  7. 2024Probing the limitations of multimodal language models for chemistry and materials researchcited by 31
  8. 2024Defending Jailbreak Prompts via In-Context Adversarial Gamecited by 27
  9. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  10. 2025Qwen Look Again: Guiding Vision-Language Reasoning Models to Re-attention Visual Informationcited by 27
  11. 2025VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videoscited by 24
  12. 2025SciVerse: Unveiling the Knowledge Comprehension and Visual Reasoning of LMMs on Multi-modal Scientific Problemscited by 22
23 more
  1. 2024ChemEval: A Comprehensive Multi-Level Chemical Evaluation for Large Language Modelscited by 18
  2. 2025HSSBench: Benchmarking Humanities and Social Sciences Ability for Multimodal Large Language Modelscited by 17
  3. 2025Dissecting Logical Reasoning in LLMs: A Fine-Grained Evaluation and Supervision Studycited by 17
  4. 2024EEE-Bench: A Comprehensive Multimodal Electrical And Electronics Engineering Benchmarkcited by 14
  5. 2024VISCO: Benchmarking Fine-Grained Critique and Correction Towards Self-Improvement in Visual Reasoningcited by 13
  6. 2025Towards LLM Agents for Earth Observationcited by 13
  7. 2025Observe-R1: Unlocking Reasoning Abilities of MLLMs with Dynamic Progressive Reinforcement Learningcited by 13
  8. 2025A Causality-aware Paradigm for Evaluating Creativity of Multimodal Large Language Modelscited by 10
  9. 2025MMSciBench: Benchmarking Language Models on Chinese Multimodal Scientific Problemscited by 10
  10. 2025MSEarth: A Multimodal Benchmark for Earth Science Phenomenon Discovery with MLLMscited by 9
  11. 2025Unveiling Chain of Step Reasoning for Vision-Language Models with Fine-grained Rewardscited by 6
  12. 2025Self-Improvement in Multimodal Large Language Models: A Surveycited by 6
  13. 2025Beyond Single-Value Metrics: Evaluating and Enhancing LLM Unlearning with Cognitive Diagnosiscited by 5
  14. 2025QCBench: Evaluating Large Language Models on Domain-Specific Quantitative Chemistrycited by 5
  15. 2025RxnBench: A Multimodal Benchmark for Evaluating Large Language Models on Chemical Reaction Understanding from Scientific Literaturecited by 3
  16. 2025GRAFT: GRaPH and Table Reasoning for Textual Alignment -- A Benchmark for Structured Instruction Following and Visual Reasoningcited by 2
  17. 2025ALDEN: Reinforcement Learning for Active Navigation and Evidence Gathering in Long Documentscited by 2
  18. 2025Argus Inspection: Do Multimodal Large Language Models Possess the Eye of Panoptes?cited by 1
  19. 2025Challenges for AI in Multimodal STEM Assessments: a Human-AI Comparisoncited by 1
  20. 2026ESC: Emotional Self-Correction for Reliable Vision-Language Modelscited by 1
  21. 2025ViSurf: Visual Supervised-and-Reinforcement Fine-Tuning for Large Vision-and-Language Modelscited by 0
  22. 2026A Survey of Multimodal Mathematical Reasoning: From Perception, Alignment to Reasoningcited by 0
  23. 2026TestHallVQA: Exploring LVLMs' Document-Level Reasoning under Redundant Contexts from Scientific Examscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025EduVidQA: Generating and Evaluating Long-form Answers to Student Questions based on Lecture Videos
  2. 2024Retrieval Meets Reasoning: Even High-school Textbook Knowledge Benefits Multimodal Reasoning
  3. 2023T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Mixed Large Language Model Signals for Science Question Answering
  4. 2025Evaluating Multimodal Large Language Models on Educational Textbook Question Answering
  5. 2024MM-PhyQA: Multimodal Physics Question-Answering With Multi-Image CoT Prompting
  6. 2025MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models
  7. 2024VisScience: An Extensive Benchmark for Evaluating K12 Educational Multi-modal Scientific Reasoning
  8. 2024CLR-Bench: Evaluating Large Language Models in College-level Reasoning