mlsanityData to 7 Oct 2026

Large Scale AI Grading of Handwritten Physics Assessments: Score Agreement and Olympiad Team Selection Outcomes

arXiv 2608.20521 ↗First posted 20 Aug 2026physics.ed-ph cs.AI cs.CYcited by 0

Not on the map. Why?

Abstract (arXiv)

Multimodal AI can read handwritten physics solutions, but high-stakes grading requires agreement with official scores and outcomes. This study evaluated GPT-5.5-based grading on 10364 scanned pages from 520 handwritten submissions by 416 unique candidates or students across three assessments: a national Physics Olympiad theory examination, the final Olympiad selection camp with theory and experiment components, and a university quantum-mechanics examination. Each submission was graded twice by AI using the official rubrics. The second round used revised page-by-page and evidence-location instructions developed after first-round disagreement analysis. During grading, AI did not see official human marks or AI--human comparisons. Total-score correlations with official marks were high (0.91--0.97). For the final Olympiad selection, AI recovered the same five-student team as official grading. The second round improved aggregate question-part agreement, especially where first-round disagreements were larger. The main difficulty remained exact partial-credit grading, especially in experimental work. Reliable AI grading therefore depends on detailed rubrics and should be used as a second reader or audit tool under examiner control.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 12 of its 28 references that are in the map. First posted on arXiv 20 Aug 2026.

Builds on

12 of 28 references.

References · 12

  1. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  2. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  3. 2020Consistent Estimators for Learning to Defer to an Expertcited by 262
  4. 2023Applying Large Language Models and Chain-of-Thought for Automatic Scoringcited by 68
  5. 2023Fine-tuning ChatGPT for Automatic Scoringcited by 50
  6. 2025Can Vision-Language Models Evaluate Handwritten Math?cited by 10
  7. 2023Automatic assessment of text-based responses in post-secondary education: A systematic reviewcited by 6
  8. 2026When VLMs 'Fix' Students: Identifying and Penalizing Over-Correction in the Evaluation of Multi-line Handwritten Math OCRcited by 3
  9. 2025Automated Grading of Students' Handwritten Graphs: A Comparison of Meta-Learning and Vision-Large Language Modelscited by 2
  10. 2025Artificial-Intelligence Grading Assistance for Handwritten Components of a Calculus Examcited by 2
  11. 2026Designing Reliable LLM-Assisted Rubric Scoring for Constructed Responses: Evidence from Physics Examscited by 2
  12. 2026Human-in-the-Loop LLM Grading for Handwritten Mathematics Assessmentscited by 1

Built on by

Not cited yet.