# LLM-as-a-Grader: Practical Insights from Large Language Model for Short-Answer and Report Evaluation

arXiv: https://arxiv.org/abs/2511.10819 · HTML: https://mlsanity.com/paper/2511.10819/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 57 references that are in the map.
- First posted on arXiv 13 Nov 2025.

## Builds on (most cited first, 9 of 9)

- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Large Language Model as an Assignment Evaluator: Insights, Feedback, and Challenges in a 1000+ Student Course](https://mlsanity.com/paper/2407.05216/) (2024)
- [Grade Like a Human: Rethinking Automated Assessment with Large Language Models](https://mlsanity.com/paper/2405.19694/) (2024)
- [How Can I Get It Right? Using GPT to Rephrase Incorrect Trainee Responses](https://mlsanity.com/paper/2405.00970/) (2024)
- [Grading Massive Open Online Courses Using Large Language Models](https://mlsanity.com/paper/2406.11102/) (2024)
- [Enhancing LLM-Based Short Answer Grading with Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.05276/) (2025)
- [How Do Students Interact with an LLM-powered Virtual Teaching Assistant in Different Educational Settings?](https://mlsanity.com/paper/2407.17429/) (2024)
- [A Zero-Shot LLM Framework for Automatic Assignment Grading in Higher Education](https://mlsanity.com/paper/2501.14305/) (2025)
- [Automated Assignment Grading with Large Language Models: Insights From a Bioinformatics Course](https://mlsanity.com/paper/2501.14499/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly explored for educational tasks such as grading, yet their alignment with human evaluation in real classrooms remains underexamined. In this study, we investigate the feasibility of using an LLM (GPT-4o) to evaluate short-answer quizzes and project reports in an undergraduate Computational Linguistics course. We collect responses from approximately 50 students across five quizzes and receive project reports from 14 teams. LLM-generated scores are compared against human evaluations conducted independently by the course teaching assistants (TAs). Our results show that GPT-4o achieves strong correlation with human graders (up to 0.98) and exact score agreement in 55% of quiz cases. For project reports, it also shows strong overall alignment with human grading, while exhibiting some variability in scoring technical, open-ended responses. We release all code and sample data to support further research on LLMs in educational assessment. This work highlights both the potential and limitations of LLM-based grading systems and contributes to advancing automated grading in real-world academic settings.