# Challenges for AI in Multimodal STEM Assessments: a Human-AI Comparison

arXiv: https://arxiv.org/abs/2507.03013 · HTML: https://mlsanity.com/paper/2507.03013/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 18 references that are in the map.
- First posted on arXiv 2 Jul 2025.

## Builds on (most cited first, 9 of 9)

- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [Large Language Models for Education: A Survey and Outlook](https://mlsanity.com/paper/2403.18105/) (2024)
- [M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models](https://mlsanity.com/paper/2306.05179/) (2023)
- [SceMQA: A Scientific College Entrance Level Multimodal Question Answering Benchmark](https://mlsanity.com/paper/2402.05138/) (2024)
- [CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery](https://mlsanity.com/paper/2406.08587/) (2024)
- [MM-PhyQA: Multimodal Physics Question-Answering With Multi-Image CoT Prompting](https://mlsanity.com/paper/2404.08704/) (2024)
- [Could ChatGPT get an Engineering Degree? Evaluating Higher Education Vulnerability to AI Assistants](https://mlsanity.com/paper/2408.11841/) (2024)
- [Performance of Large Language Models in a Computer Science Degree Program](https://mlsanity.com/paper/2308.02432/) (2023)
- [Survey of Natural Language Processing for Education: Taxonomy, Systematic Review, and Future Trends](https://mlsanity.com/paper/2401.07518/) (2024)

## Built on by (most cited first, 1 of 1)

- [Incentives Of EdTech: A Systematic Review Of EduNLP Research](https://mlsanity.com/paper/2606.13691/) (2026)

## Abstract (arXiv)

Generative AI systems have rapidly advanced, with multimodal input capabilities enabling reasoning beyond text-based tasks. In education, these advancements could influence assessment design and question answering, presenting both opportunities and challenges. To investigate these effects, we introduce a high-quality dataset of 201 university-level STEM questions, manually annotated with features such as image type, role, problem complexity, and question format. Our study analyzes how these features affect generative AI performance compared to students. We evaluate four model families with five prompting strategies, comparing results to the average of 546 student responses per question. Although the best model correctly answers on average 58.5 % of the questions using majority vote aggregation, human participants consistently outperform AI on questions involving visual components. Interestingly, human performance remains stable across question features but varies by subject, whereas AI performance is susceptible to both subject matter and question features. Finally, we provide actionable insights for educators, demonstrating how question design can enhance academic integrity by leveraging features that challenge current AI systems without increasing the cognitive burden for students.