# A LLM-Powered Automatic Grading Framework with Human-Level Guidelines Optimization

arXiv: https://arxiv.org/abs/2410.02165 · HTML: https://mlsanity.com/paper/2410.02165/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 49 references that are in the map.
- First posted on arXiv 3 Oct 2024.

## Builds on (most cited first, 28 of 28)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies](https://mlsanity.com/paper/2308.03188/) (2023)
- [Beam Search Strategies for Neural Machine Translation](https://mlsanity.com/paper/1702.01806/) (2017)
- [Alignment of Language Agents](https://mlsanity.com/paper/2103.14659/) (2021)
- [Applying Large Language Models and Chain-of-Thought for Automatic Scoring](https://mlsanity.com/paper/2312.03748/) (2023)
- [Fine-tuning ChatGPT for Automatic Scoring](https://mlsanity.com/paper/2310.10072/) (2023)
- [Are Large Language Models Good Prompt Optimizers?](https://mlsanity.com/paper/2402.02101/) (2024)
- [A Chain-of-Thought Prompting Approach with LLMs for Evaluating Students' Formative Assessment Responses in Science](https://mlsanity.com/paper/2403.14565/) (2024)
- [Task Facet Learning: A Structured Approach to Prompt Optimization](https://mlsanity.com/paper/2406.10504/) (2024)
- [Grade Like a Human: Rethinking Automated Assessment with Large Language Models](https://mlsanity.com/paper/2405.19694/) (2024)
- [Misconfidence-based Demonstration Selection for LLM In-Context Learning](https://mlsanity.com/paper/2401.06301/) (2024)
- [Automatic Short Answer Grading and Feedback Using Text Mining Methods](https://mlsanity.com/paper/1807.10543/) (2018)
- [Can Large Language Models Make the Grade? An Empirical Study Evaluating LLMs Ability to Mark Short Answer Questions in K-12 Education](https://mlsanity.com/paper/2405.02985/) (2024)
- [Content Knowledge Identification with Multi-Agent Large Language Models (LLMs)](https://mlsanity.com/paper/2404.07960/) (2024)
- [Explainable Automatic Grading with Neural Additive Models](https://mlsanity.com/paper/2405.00489/) (2024)

## Built on by (most cited first, 16 of 16)

- [Agent-as-a-Judge](https://mlsanity.com/paper/2601.05111/) (2026)
- [LLM-based Automated Grading with Human-in-the-Loop](https://mlsanity.com/paper/2504.05239/) (2025)
- [RefGrader: Automated Grading of Mathematical Competition Proofs using Agentic Workflows](https://mlsanity.com/paper/2510.09021/) (2025)
- [Pensieve Grader: An AI-Powered, Ready-to-Use Platform for Effortless Handwritten STEM Grading](https://mlsanity.com/paper/2507.01431/) (2025)
- [Evidence-Decision-Feedback: Theory-Driven Adaptive Scaffolding for LLM Agents](https://mlsanity.com/paper/2602.01415/) (2026)
- [Confusion-Aware Rubric Optimization for LLM-based Automated Grading](https://mlsanity.com/paper/2603.00451/) (2026)
- [Don't Overthink It: Inter-Rollout Action Agreement as a Free Adaptive-Compute Signal for LLM Agents](https://mlsanity.com/paper/2604.08369/) (2026)
- [On the effectiveness of LLMs for automatic grading of open-ended questions in Spanish](https://mlsanity.com/paper/2503.18072/) (2025)
- [A LLM-Driven Multi-Agent Systems for Professional Development of Mathematics Teachers](https://mlsanity.com/paper/2507.05292/) (2025)
- [Implicit Grading Bias in Large Language Models: How Writing Style Affects Automated Assessment Across Math, Programming, and Essay Tasks](https://mlsanity.com/paper/2603.18765/) (2026)
- ["**Important** You should give me full credits!": Exploring Prompt Injection Attacks on LLM-Based Automatic Grading Systems](https://mlsanity.com/paper/2606.03090/) (2026)
- [AI Meets Mathematics Education: A Case Study on Supporting an Instructor in a Large Mathematics Class with Context-Aware AI](https://mlsanity.com/paper/2603.26679/) (2026)
- [From Tool to Teammate: LLM Coding Agents as Collaborative Partners for Behavioral Labeling in Educational Dialogue Analysis](https://mlsanity.com/paper/2603.27440/) (2026)
- [Learnable Assessment Skills for LLM-based Automated Scoring: Rubric Construction via Iterative Optimization](https://mlsanity.com/paper/2605.29274/) (2026)
- [A Theory-Guided LLM Pedagogical Agent for STEM+C Scaffolding Without Over-Reliance](https://mlsanity.com/paper/2605.30539/) (2026)
- [LLMs as Teaching Assistants for Mathematics Exam Grading: Reliability, and Practical Usability](https://mlsanity.com/paper/2607.01247/) (2026)

## Abstract (arXiv)

Open-ended short-answer questions (SAGs) have been widely recognized as a powerful tool for providing deeper insights into learners' responses in the context of learning analytics (LA). However, SAGs often present challenges in practice due to the high grading workload and concerns about inconsistent assessments. With recent advancements in natural language processing (NLP), automatic short-answer grading (ASAG) offers a promising solution to these challenges. Despite this, current ASAG algorithms are often limited in generalizability and tend to be tailored to specific questions. In this paper, we propose a unified multi-agent ASAG framework, GradeOpt, which leverages large language models (LLMs) as graders for SAGs. More importantly, GradeOpt incorporates two additional LLM-based agents - the reflector and the refiner - into the multi-agent system. This enables GradeOpt to automatically optimize the original grading guidelines by performing self-reflection on its errors. Through experiments on a challenging ASAG task, namely the grading of pedagogical content knowledge (PCK) and content knowledge (CK) questions, GradeOpt demonstrates superior performance in grading accuracy and behavior alignment with human graders compared to representative baselines. Finally, comprehensive ablation studies confirm the effectiveness of the individual components designed in GradeOpt.