# Performance of the Pre-Trained Large Language Model GPT-4 on Automated Short Answer Grading

arXiv: https://arxiv.org/abs/2309.09338 · HTML: https://mlsanity.com/paper/2309.09338/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 40 references that are in the map.
- First posted on arXiv 17 Sep 2023.

## Builds on (most cited first, 3 of 3)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Survey on Automated Short Answer Grading with Deep Learning: from Word Embeddings to Transformers](https://mlsanity.com/paper/2204.03503/) (2022)

## Built on by (most cited first, 15 of 15)

- [Automated Long Answer Grading with RiceChem Dataset](https://mlsanity.com/paper/2404.14316/) (2024)
- [Can LLMs Grade Short-Answer Reading Comprehension Questions : An Empirical Study with a Novel Dataset](https://mlsanity.com/paper/2310.18373/) (2023)
- [Artificial-Intelligence Grading Assistance for Handwritten Components of a Calculus Exam](https://mlsanity.com/paper/2510.05162/) (2025)
- [EvalYaks: Instruction Tuning Datasets and LoRA Fine-tuned Models for Automated Scoring of CEFR B2 Speaking Assessment Transcripts](https://mlsanity.com/paper/2408.12226/) (2024)
- [Concept-based Rubrics Improve LLM Formative Assessment and Data Synthesis](https://mlsanity.com/paper/2504.03877/) (2025)
- [Overview of AI Grading of Physics Olympiad Exams](https://mlsanity.com/paper/2505.02121/) (2025)
- [Comparing Human and AI Rater Effects Using the Many-Facet Rasch Model](https://mlsanity.com/paper/2505.18486/) (2025)
- [FEAT: A Preference Feedback Dataset through a Cost-Effective Auto-Generation and Labeling Framework for English AI Tutoring](https://mlsanity.com/paper/2506.19325/) (2025)
- [Learning to Love Edge Cases in Formative Math Assessment: Using the AMMORE Dataset and Chain-of-Thought Prompting to Improve Grading Accuracy](https://mlsanity.com/paper/2409.17904/) (2024)
- [Leveraging AI Graders for Missing Score Imputation to Achieve Accurate Ability Estimation in Constructed-Response Tests](https://mlsanity.com/paper/2506.20119/) (2025)
- [AI-Enabled grading with near-domain data for scaling feedback with human-level accuracy](https://mlsanity.com/paper/2512.04113/) (2025)
- [Grading Handwritten Engineering Exams with Multimodal Large Language Models](https://mlsanity.com/paper/2601.00730/) (2026)
- [S-GRADES -- Studying Generalization of Student Response Assessments in Diverse Evaluative Settings](https://mlsanity.com/paper/2603.10233/) (2026)
- [Quality-Conditioned Agreement in Automated Short Answer Scoring: Mid-Range Degradation and the Impact of Task-Specific Adaptation](https://mlsanity.com/paper/2605.07647/) (2026)
- [Bounding Boxes to Improve Small Language Model Performance on Vision-Based Grading Tasks](https://mlsanity.com/paper/2607.18767/) (2026)

## Abstract (arXiv)

Automated Short Answer Grading (ASAG) has been an active area of machine-learning research for over a decade. It promises to let educators grade and give feedback on free-form responses in large-enrollment courses in spite of limited availability of human graders. Over the years, carefully trained models have achieved increasingly higher levels of performance. More recently, pre-trained Large Language Models (LLMs) emerged as a commodity, and an intriguing question is how a general-purpose tool without additional training compares to specialized models. We studied the performance of GPT-4 on the standard benchmark 2-way and 3-way datasets SciEntsBank and Beetle, where in addition to the standard task of grading the alignment of the student answer with a reference answer, we also investigated withholding the reference answer. We found that overall, the performance of the pre-trained general-purpose GPT-4 LLM is comparable to hand-engineered models, but worse than pre-trained LLMs that had specialized training.