mlsanityData to 7 Oct 2026

LLM-as-a-judge validity is strongly task-dependent across physics assessment formats

arXiv 2603.14732 ↗First posted 16 Mar 2026physics.ed-ph cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

As large language models (LLMs) are increasingly considered for automated assessment and feedback, understanding when LLM marking is valid is essential. We evaluate LLM-as-a-judge marking across four settings spanning three assessment formats - structured questions, written essays, and scientific plots - comparing GPT-5.2, Grok 4.1, Claude Opus 4.5, DeepSeek-V3.2, Gemini 3 Pro, and committee aggregations against human markers under blind, solution-provided, false-solution, and anchored conditions. We distinguish absolute accuracy from rank-order agreement, since a marking system can match the distribution of human marks while failing to order responses by quality. Across task types, performance is sharply task-dependent. For blind university exam questions (n=771) and secondary and university structured questions (n=1151), models show robust rank-order agreement with human markers (Spearman ρ > 0.6), with official solutions reducing error and strengthening agreement. False solutions degrade absolute accuracy, showing that models defer to provided references, but leave rank-ordering intact. Essay marking behaves fundamentally differently. Across n=55 scripts, each containing five essays (n=275 essays total), blind AI marking is harsher and more variable than human marking and adding marking guidance does not improve rank-order agreement. Anchored exemplars shift the AI mean close to the human mean and compress variance below the human standard deviation, but rank-order agreement remains near-zero. For code-based plot elements (n=1400), models achieve high rank-order agreement (ρ > 0.83) with near-linear calibration. LLM marking validity depended strongly on the assessment task, and this held for every contemporary model tested. The results also show that the reliability of the human benchmark constrains the claims that can be made about AI-human agreement.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 14 of its 57 references that are in the map. First posted on arXiv 16 Mar 2026.

Builds on

14 of 57 references.

References · 14

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  5. 2024A Survey on LLM-as-a-Judgecited by 1,384
  6. 2023Applying Large Language Models and Chain-of-Thought for Automatic Scoringcited by 68
  7. 2024Anchoring Bias in Large Language Models: An Experimental Studycited by 37
  8. 2025DrawEduMath: Evaluating Vision Language Models with Expert-Annotated Students' Hand-Drawn Math Imagescited by 17
  9. 2024Taking the Next Step with Generative Artificial Intelligence: The Transformative Role of Multimodal Large Language Models in Science Educationcited by 15
  10. 2023NERIF: GPT-4V for Automatic Scoring of Drawn Modelscited by 13
  11. 2024A comparison of Human, GPT-3.5, and GPT-4 Performance in a University-Level Coding Coursecited by 10
  12. 2025Can Vision-Language Models Evaluate Handwritten Math?cited by 10
2 more
  1. 2025Evaluating GPT- and Reasoning-based Large Language Models on Physics Olympiad Problems: Surpassing Human Performance and Implications for Educational Assessmentcited by 7
  2. 2025Multilingual Performance of a Multimodal Artificial Intelligence System on Multisubject Physics Concept Inventoriescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026The Answer Is Not the Argumentcited by 0