mlsanityData to 7 Oct 2026

Can Vision-Language Models Evaluate Handwritten Math?

arXiv 2501.07244 ↗First posted 13 Jan 2025cs.CV cs.CLcited by 10

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Recent advancements in Vision-Language Models (VLMs) have opened new possibilities in automatic grading of handwritten student responses, particularly in mathematics. However, a comprehensive study to test the ability of VLMs to evaluate and reason over handwritten content remains absent. To address this gap, we introduce FERMAT, a benchmark designed to assess the ability of VLMs to detect, localize and correct errors in handwritten mathematical content. FERMAT spans four key error dimensions - computational, conceptual, notational, and presentation - and comprises over 2,200 handwritten math solutions derived from 609 manually curated problems from grades 7-12 with intentionally introduced perturbations. Using FERMAT we benchmark nine VLMs across three tasks: error detection, localization, and correction. Our results reveal significant shortcomings in current VLMs in reasoning over handwritten text, with Gemini-1.5-Pro achieving the highest error correction rate (77%). We also observed that some models struggle with processing handwritten content, as their accuracy improves when handwritten inputs are replaced with printed text or images. These findings highlight the limitations of current VLMs and reveal new avenues for improvement. We release FERMAT and all the associated resources in the open-source to drive further research.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 43 references that are in the map. First posted on arXiv 13 Jan 2025.

Builds on

30 of 43 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detection6 Oct
  2. 2024EXAMS-V: A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Models15 Mar
  3. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset22 Feb
  4. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems21 Feb
  5. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  6. 2023Vision-Language Models for Vision Tasks: A Survey3 Apr

Other references · 24

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  7. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  8. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  9. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  10. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  11. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  12. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
12 more
  1. 2024Large Language Models for Education: A Survey and Outlookcited by 185
  2. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  3. 2023Learning From Mistakes Makes LLM Better Reasonercited by 97
  4. 2023Bridging the Novice-Expert Gap via Models of Decision-Making: A Case Study on Remediating Math Mistakescited by 51
  5. 2024Evaluating LLMs at Detecting Errors in LLM Responsescited by 49
  6. 2024Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklistcited by 48
  7. 2024Evaluating Mathematical Reasoning of Large Language Models: A Focus on Error Identification and Correctioncited by 45
  8. 2024Finding Blind Spots in Evaluator LLMs with Interpretable Checklistscited by 32
  9. 2024Knowledge Graphs as Context Sources for LLM-Based Explanations of Learning Recommendationscited by 31
  10. 2024Automated Educational Question Generation at Different Bloom's Skill Levels using Large Language Models: Strategies and Evaluationcited by 29
  11. 2024Bringing Generative AI to Adaptive Learning in Educationcited by 24
  12. 2023Analyzing the Quality of Submissions in Online Programming Coursescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 7.

10 papers.

  1. 2026Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understandingcited by 6
  2. 2025VEHME: A Vision-Language Model For Evaluating Handwritten Mathematics Expressionscited by 4
  3. 2025Uni-MuMER: Unified Multi-Task Fine-Tuning of Vision-Language Model for Handwritten Mathematical Expression Recognitioncited by 3
  4. 2026When VLMs 'Fix' Students: Identifying and Penalizing Over-Correction in the Evaluation of Multi-line Handwritten Math OCRcited by 3
  5. 2025Rice-VL: Evaluating Vision-Language Models for Cultural Understanding Across ASEAN Countriescited by 1
  6. 2026The Aftermath of DrawEduMath: Vision Language Models Underperform with Struggling Students and Misdiagnose Errorscited by 1
  7. 2026LLM-as-a-judge validity is strongly task-dependent across physics assessment formatscited by 1
  8. 2026EDU-CIRCUIT-HW: Evaluating Multimodal Large Language Models on Real-World University-Level STEM Student Handwritten Solutionscited by 0
  9. 2026Exploring Vision-Language Models for Online Signature Verification: A Zero-Shot Capability Studycited by 0
  10. 2026Large Scale AI Grading of Handwritten Physics Assessments: Score Agreement and Olympiad Team Selection Outcomescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Can MLLMs Read Students' Minds? Unpacking Multimodal Error Analysis in Handwritten Math
  2. 2025DrawEduMath: Evaluating Vision Language Models with Expert-Annotated Students' Hand-Drawn Math Images
  3. 2025Evaluating Vision-Language and Large Language Models for Automated Student Assessment in Indonesian Classrooms
  4. 2025Benchmarking Large Language Models for Diagnosing Students' Cognitive Skills from Handwritten Math Work
  5. 2025CHECK-MAT: Checking Hand-Written Mathematical Answers for the Russian Unified State Exam
  6. 2026Evaluating AI Grading on Real-World Handwritten College Mathematics: A Large-Scale Study Toward a Benchmark
  7. 2025GSM8K-V: Can Vision Language Models Solve Grade School Math Word Problems in Visual Contexts
  8. 2025Seeing the Big Picture: Evaluating Multimodal LLMs' Ability to Interpret and Grade Handwritten Student Work