# DrawEduMath: Evaluating Vision Language Models with Expert-Annotated Students' Hand-Drawn Math Images

arXiv: https://arxiv.org/abs/2501.14877 · HTML: https://mlsanity.com/paper/2501.14877/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 19 of its 53 references that are in the map.
- First posted on arXiv 24 Jan 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 19 of 19)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?](https://mlsanity.com/paper/2403.14624/) (2024)
- [Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning](https://mlsanity.com/paper/2105.04165/) (2021)
- [GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning](https://mlsanity.com/paper/2105.14517/) (2021)
- [Connecting Vision and Language with Localized Narratives](https://mlsanity.com/paper/1912.03098/) (2019)
- [All You May Need for VQA are Image Captions](https://mlsanity.com/paper/2205.01883/) (2022)
- [Algorithmic Fairness in Education](https://mlsanity.com/paper/2007.05443/) (2020)
- [A large language model-assisted education tool to provide feedback on open-ended responses](https://mlsanity.com/paper/2308.02439/) (2023)
- [Asking questions on handwritten document collections](https://mlsanity.com/paper/2110.00711/) (2021)
- [Beyond "Fairness:" Structural (In)justice Lenses on AI for Education](https://mlsanity.com/paper/2105.08847/) (2021)

## Built on by (most cited first, 17 of 17)

- [Learning to Make MISTAKEs: Modeling Incorrect Student Thinking And Key Errors](https://mlsanity.com/paper/2510.11502/) (2025)
- [HoneyBee: Data Recipes for Vision-Language Reasoners](https://mlsanity.com/paper/2510.12225/) (2025)
- [MMSciBench: Benchmarking Language Models on Chinese Multimodal Scientific Problems](https://mlsanity.com/paper/2503.01891/) (2025)
- [VEHME: A Vision-Language Model For Evaluating Handwritten Mathematics Expressions](https://mlsanity.com/paper/2510.22798/) (2025)
- [SketchMind: A Multi-Agent Cognitive Framework for Assessing Student-Drawn Scientific Sketches](https://mlsanity.com/paper/2507.22904/) (2025)
- [Automated Grading of Students' Handwritten Graphs: A Comparison of Meta-Learning and Vision-Large Language Models](https://mlsanity.com/paper/2507.03056/) (2025)
- [Benchmarking Large Language Models for Diagnosing Students' Cognitive Skills from Handwritten Math Work](https://mlsanity.com/paper/2504.00843/) (2025)
- [Misconception Diagnosis From Student-Tutor Dialogue: Generate, Retrieve, Rerank](https://mlsanity.com/paper/2602.02414/) (2026)
- [The Aftermath of DrawEduMath: Vision Language Models Underperform with Struggling Students and Misdiagnose Errors](https://mlsanity.com/paper/2603.00925/) (2026)
- [LLM-as-a-judge validity is strongly task-dependent across physics assessment formats](https://mlsanity.com/paper/2603.14732/) (2026)
- [Exploring Agentic Workflows for Generating High Quality Math Visual Aids](https://mlsanity.com/paper/2607.09839/) (2026)
- [Seeing the Big Picture: Evaluating Multimodal LLMs' Ability to Interpret and Grade Handwritten Student Work](https://mlsanity.com/paper/2510.05538/) (2025)
- [Grading Handwritten Engineering Exams with Multimodal Large Language Models](https://mlsanity.com/paper/2601.00730/) (2026)
- [EDU-CIRCUIT-HW: Evaluating Multimodal Large Language Models on Real-World University-Level STEM Student Handwritten Solutions](https://mlsanity.com/paper/2602.00095/) (2026)
- [ConceptKT: A Benchmark for Concept-Level Deficiency Prediction in Knowledge Tracing](https://mlsanity.com/paper/2603.24073/) (2026)
- [Can Vision Language Models Be Adaptive in Mathematics Education? A Learner Model-based Rubric Study](https://mlsanity.com/paper/2605.16011/) (2026)
- [MIDAL: A Dataset of Math Image Descriptions for Accessible Learning](https://mlsanity.com/paper/2608.00868/) (2026)

## Abstract (arXiv)

In real-world settings, vision language models (VLMs) should robustly handle naturalistic, noisy visual content as well as domain-specific language and concepts. For example, K-12 educators using digital learning platforms may need to examine and provide feedback across many images of students' math work. To assess the potential of VLMs to support educators in settings like this one, we introduce DrawEduMath, an English-language dataset of 2,030 images of students' handwritten responses to K-12 math problems. Teachers provided detailed annotations, including free-form descriptions of each image and 11,661 question-answer (QA) pairs. These annotations capture a wealth of pedagogical insights, ranging from students' problem-solving strategies to the composition of their drawings, diagrams, and writing. We evaluate VLMs on teachers' QA pairs, as well as 44,362 synthetic QA pairs derived from teachers' descriptions using language models (LMs). We show that even state-of-the-art VLMs leave much room for improvement on DrawEduMath questions. We also find that synthetic QAs, though imperfect, can yield similar model rankings as teacher-written QAs. We release DrawEduMath to support the evaluation of VLMs' abilities to reason mathematically over images gathered with educational contexts in mind.