# Towards LLM-based Autograding for Short Textual Answers

arXiv: https://arxiv.org/abs/2309.11508 · HTML: https://mlsanity.com/paper/2309.11508/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 38 references that are in the map.
- First posted on arXiv 9 Sep 2023.

## Builds on (most cited first, 5 of 5)

- [Mathematical Capabilities of ChatGPT](https://mlsanity.com/paper/2301.13867/) (2023)
- [A Categorical Archive of ChatGPT Failures](https://mlsanity.com/paper/2302.03494/) (2023)
- [Practical and Ethical Challenges of Large Language Models in Education: A Systematic Scoping Review](https://mlsanity.com/paper/2303.13379/) (2023)
- [Survey on Automated Short Answer Grading with Deep Learning: from Word Embeddings to Transformers](https://mlsanity.com/paper/2204.03503/) (2022)
- [Towards Trustworthy AutoGrading of Short, Multi-lingual, Multi-type Answers](https://mlsanity.com/paper/2201.03425/) (2022)

## Built on by (most cited first, 20 of 20)

- [The Counterfeit Conundrum: Can Code Language Models Grasp the Nuances of Their Incorrect Generations?](https://mlsanity.com/paper/2402.19475/) (2024)
- [LLM-based Automated Grading with Human-in-the-Loop](https://mlsanity.com/paper/2504.05239/) (2025)
- [ElicitationGPT: Text Elicitation Mechanisms via Language Models](https://mlsanity.com/paper/2406.09363/) (2024)
- [Skewed Score: A statistical framework to assess autograders](https://mlsanity.com/paper/2507.03772/) (2025)
- [Content Knowledge Identification with Multi-Agent Large Language Models (LLMs)](https://mlsanity.com/paper/2404.07960/) (2024)
- [Beyond Scores: A Modular RAG-Based System for Automatic Short Answer Scoring with Feedback](https://mlsanity.com/paper/2409.20042/) (2024)
- [Co-designing Large Language Model Tools for Project-Based Learning with K12 Educators](https://mlsanity.com/paper/2502.09799/) (2025)
- [Can LLMs Grade Short-Answer Reading Comprehension Questions : An Empirical Study with a Novel Dataset](https://mlsanity.com/paper/2310.18373/) (2023)
- [Knowledge Distillation of LLM for Automatic Scoring of Science Education Assessments](https://mlsanity.com/paper/2312.15842/) (2023)
- [SteLLA: A Structured Grading System Using LLMs with RAG](https://mlsanity.com/paper/2501.09092/) (2025)
- [Aligned Textual Scoring Rules](https://mlsanity.com/paper/2507.06221/) (2025)
- [EvalYaks: Instruction Tuning Datasets and LoRA Fine-tuned Models for Automated Scoring of CEFR B2 Speaking Assessment Transcripts](https://mlsanity.com/paper/2408.12226/) (2024)
- [On the effectiveness of LLMs for automatic grading of open-ended questions in Spanish](https://mlsanity.com/paper/2503.18072/) (2025)
- [Fine-tuning for Better Few Shot Prompting: An Empirical Comparison for Short Answer Grading](https://mlsanity.com/paper/2508.04063/) (2025)
- [Towards Human-Like Grading: A Unified LLM-Enhanced Framework for Subjective Question Evaluation](https://mlsanity.com/paper/2510.07912/) (2025)
- [Learning to Love Edge Cases in Formative Math Assessment: Using the AMMORE Dataset and Chain-of-Thought Prompting to Improve Grading Accuracy](https://mlsanity.com/paper/2409.17904/) (2024)
- [Thematic and Task-Based Categorization of K-12 GenAI Usages with Hierarchical Topic Modeling](https://mlsanity.com/paper/2508.09997/) (2025)
- [Seeing the Big Picture: Evaluating Multimodal LLMs' Ability to Interpret and Grade Handwritten Student Work](https://mlsanity.com/paper/2510.05538/) (2025)
- [AI-Enabled grading with near-domain data for scaling feedback with human-level accuracy](https://mlsanity.com/paper/2512.04113/) (2025)
- [LLMs as Teaching Assistants for Mathematics Exam Grading: Reliability, and Practical Usability](https://mlsanity.com/paper/2607.01247/) (2026)

## Abstract (arXiv)

Grading exams is an important, labor-intensive, subjective, repetitive, and frequently challenging task. The feasibility of autograding textual responses has greatly increased thanks to the availability of large language models (LLMs) such as ChatGPT and the substantial influx of data brought about by digitalization. However, entrusting AI models with decision-making roles raises ethical considerations, mainly stemming from potential biases and issues related to generating false information. Thus, in this manuscript, we provide an evaluation of a large language model for the purpose of autograding, while also highlighting how LLMs can support educators in validating their grading procedures. Our evaluation is targeted towards automatic short textual answers grading (ASAG), spanning various languages and examinations from two distinct courses. Our findings suggest that while "out-of-the-box" LLMs provide a valuable tool to provide a complementary perspective, their readiness for independent automated grading remains a work in progress, necessitating human oversight.