# EDU-CIRCUIT-HW: Evaluating Multimodal Large Language Models on Real-World University-Level STEM Student Handwritten Solutions

arXiv: https://arxiv.org/abs/2602.00095 · HTML: https://mlsanity.com/paper/2602.00095/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 11 of its 60 references that are in the map.
- First posted on arXiv 23 Jan 2026.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)
- Mechanistic interpretability / [Circuits and mechanistic analysis](https://mlsanity.com/mechinterp/circuits-and-mechanistic-analysis/)

## Builds on (most cited first, 11 of 11)

- [MathBERT: A Pre-Trained Model for Mathematical Formula Understanding](https://mlsanity.com/paper/2105.00377/) (2021)
- [DrawEduMath: Evaluating Vision Language Models with Expert-Annotated Students' Hand-Drawn Math Images](https://mlsanity.com/paper/2501.14877/) (2025)
- [CIRCUIT: A Benchmark for Circuit Interpretation and Reasoning Capabilities of LLMs](https://mlsanity.com/paper/2502.07980/) (2025)
- [Can Vision-Language Models Evaluate Handwritten Math?](https://mlsanity.com/paper/2501.07244/) (2025)
- [Hand-Drawn Electrical Circuit Recognition using Object Detection and Node Recognition](https://mlsanity.com/paper/2106.11559/) (2021)
- [Benchmarking Large Language Models on Homework Assessment in Circuit Analysis](https://mlsanity.com/paper/2506.06390/) (2025)
- [Pensieve Grader: An AI-Powered, Ready-to-Use Platform for Effortless Handwritten STEM Grading](https://mlsanity.com/paper/2507.01431/) (2025)
- [WIP: Large Language Model-Enhanced Smart Tutor for Undergraduate Circuit Analysis](https://mlsanity.com/paper/2506.08962/) (2025)
- [CHECK-MAT: Checking Hand-Written Mathematical Answers for the Russian Unified State Exam](https://mlsanity.com/paper/2507.22958/) (2025)
- [Enhancing Large Language Models for Automated Homework Assessment in Undergraduate Circuit Analysis](https://mlsanity.com/paper/2511.18221/) (2025)
- [Enhancing Large Language Model-Based Systems for End-to-End Circuit Analysis Problem Solving](https://mlsanity.com/paper/2512.10159/) (2025)

## Abstract (arXiv)

Multimodal Large Language Models (MLLMs) hold significant promise for revolutionizing traditional education and reducing teachers' workload. However, accurately interpreting unconstrained STEM student handwritten solutions with intertwined mathematical formulas, diagrams, and textual reasoning poses a significant challenge due to the lack of authentic and domain-specific benchmarks. Additionally, current evaluation paradigms predominantly rely on the outcomes of downstream tasks (e.g., auto-grading), which often probe only a subset of the recognized content, thereby failing to capture the MLLMs' understanding of complex handwritten logic as a whole. To bridge this gap, we release EDU-CIRCUIT-HW, a dataset consisting of 1,300+ authentic student handwritten solutions from a university-level STEM course. Utilizing the expert-verified verbatim transcriptions and grading reports of student solutions, we simultaneously evaluate various MLLMs' upstream recognition fidelity and downstream auto-grading performance. Our evaluation uncovers an astonishing scale of latent failures within MLLM-recognized student handwritten content, highlighting the models' insufficient reliability for auto-grading and other understanding-oriented applications in high-stakes educational settings. As a potential solution, we present a case study demonstrating that leveraging identified error patterns to preemptively detect and correct recognition errors, while requiring only minimal human intervention (e.g., routing 3.3% of assignments to human graders and the remainder to the GPT-5.1 grader), can effectively enhance the robustness of the deployed AI-enabled grading system. Code and dataset are available in this GitHub repo: https://gt-learning-innovation.github.io/CIRCUIT_EDU_HW_ACL.