# Using Large Language Models for Automated Grading of Student Writing about Science

arXiv: https://arxiv.org/abs/2412.18719 · HTML: https://mlsanity.com/paper/2412.18719/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 81 references that are in the map.
- First posted on arXiv 25 Dec 2024.

## Builds on (most cited first, 7 of 7)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models](https://mlsanity.com/paper/2303.10420/) (2023)
- [Time Travel in LLMs: Tracing Data Contamination in Large Language Models](https://mlsanity.com/paper/2308.08493/) (2023)
- [Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Models](https://mlsanity.com/paper/2311.06233/) (2023)
- [Can Large Language Models Discern Evidence for Scientific Hypotheses? Case Studies in the Social Sciences](https://mlsanity.com/paper/2309.06578/) (2023)
- [MAPLE: Micro Analysis of Pairwise Language Evolution for Few-Shot Claim Verification](https://mlsanity.com/paper/2401.16282/) (2024)
- [Large Language Models As MOOCs Graders](https://mlsanity.com/paper/2402.03776/) (2024)

## Built on by (most cited first, 12 of 12)

- [CoTAL: Human-in-the-Loop Prompt Engineering for Generalizable Formative Assessment Scoring and Feedback](https://mlsanity.com/paper/2504.02323/) (2025)
- [Automated Assignment Grading with Large Language Models: Insights From a Bioinformatics Course](https://mlsanity.com/paper/2501.14499/) (2025)
- [ChatGPT for automated grading of short answer questions in mechanical ventilation](https://mlsanity.com/paper/2505.04645/) (2025)
- [Reliable Decision Support with LLMs: A Framework for Evaluating Consistency in Binary Text Classification Applications](https://mlsanity.com/paper/2505.14918/) (2025)
- [RubiSCoT: A Framework for AI-Supported Academic Assessment](https://mlsanity.com/paper/2510.17309/) (2025)
- [Scaling Equitable Reflection Assessment in Education via Large Language Models and Role-Based Feedback Agents](https://mlsanity.com/paper/2511.11772/) (2025)
- [GradingAttack: Exposing Security Vulnerabilities in LLM Based Educational Grading Agents](https://mlsanity.com/paper/2602.00979/) (2026)
- [Toward Self-Driving Universities: Can Universities Drive Themselves with Agentic AI?](https://mlsanity.com/paper/2602.18461/) (2026)
- [CHiL(L)Grader: Calibrated Human-in-the-Loop Short-Answer Grading](https://mlsanity.com/paper/2603.11957/) (2026)
- [When AI Meets Early Childhood Education: Large Language Models as Assessment Teammates in Chinese Preschools](https://mlsanity.com/paper/2603.24389/) (2026)
- [Learnable Assessment Skills for LLM-based Automated Scoring: Rubric Construction via Iterative Optimization](https://mlsanity.com/paper/2605.29274/) (2026)
- [LLMs as Teaching Assistants for Mathematics Exam Grading: Reliability, and Practical Usability](https://mlsanity.com/paper/2607.01247/) (2026)

## Abstract (arXiv)

Assessing writing in large classes for formal or informal learners presents a significant challenge. Consequently, most large classes, particularly in science, rely on objective assessment tools such as multiple-choice quizzes, which have a single correct answer. The rapid development of AI has introduced the possibility of using large language models (LLMs) to evaluate student writing. An experiment was conducted using GPT-4 to determine if machine learning methods based on LLMs can match or exceed the reliability of instructor grading in evaluating short writing assignments on topics in astronomy. The audience consisted of adult learners in three massive open online courses (MOOCs) offered through Coursera. One course was on astronomy, the second was on astrobiology, and the third was on the history and philosophy of astronomy. The results should also be applicable to non-science majors in university settings, where the content and modes of evaluation are similar. The data comprised answers from 120 students to 12 questions across the three courses. GPT-4 was provided with total grades, model answers, and rubrics from an instructor for all three courses. In addition to evaluating how reliably the LLM reproduced instructor grades, the LLM was also tasked with generating its own rubrics. Overall, the LLM was more reliable than peer grading, both in aggregate and by individual student, and approximately matched instructor grades for all three online courses. The implication is that LLMs may soon be used for automated, reliable, and scalable grading of student science writing.