mlsanityData to 14 Oct 2026

A LLM-Powered Automatic Grading Framework with Human-Level Guidelines Optimization

arXiv 2410.02165 ↗First posted 3 Oct 2024cs.AI cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

Open-ended short-answer questions (SAGs) have been widely recognized as a powerful tool for providing deeper insights into learners' responses in the context of learning analytics (LA). However, SAGs often present challenges in practice due to the high grading workload and concerns about inconsistent assessments. With recent advancements in natural language processing (NLP), automatic short-answer grading (ASAG) offers a promising solution to these challenges. Despite this, current ASAG algorithms are often limited in generalizability and tend to be tailored to specific questions. In this paper, we propose a unified multi-agent ASAG framework, GradeOpt, which leverages large language models (LLMs) as graders for SAGs. More importantly, GradeOpt incorporates two additional LLM-based agents - the reflector and the refiner - into the multi-agent system. This enables GradeOpt to automatically optimize the original grading guidelines by performing self-reflection on its errors. Through experiments on a challenging ASAG task, namely the grading of pedagogical content knowledge (PCK) and content knowledge (CK) questions, GradeOpt demonstrates superior performance in grading accuracy and behavior alignment with human graders compared to representative baselines. Finally, comprehensive ablation studies confirm the effectiveness of the individual components designed in GradeOpt.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 28 of its 49 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

28 of 49 references.

References · 28

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  7. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  8. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  9. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  10. 2016An overview of gradient descent optimization algorithmscited by 1,033
  11. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  12. 2019How Can We Know What Language Models Know?cited by 959
16 more
  1. 2023Large Language Models as Optimizerscited by 864
  2. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 653
  3. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 238
  4. 2017Beam Search Strategies for Neural Machine Translationcited by 232
  5. 2021Alignment of Language Agentscited by 168
  6. 2023Applying Large Language Models and Chain-of-Thought for Automatic Scoringcited by 69
  7. 2023Fine-tuning ChatGPT for Automatic Scoringcited by 51
  8. 2024Are Large Language Models Good Prompt Optimizers?cited by 37
  9. 2024A Chain-of-Thought Prompting Approach with LLMs for Evaluating Students' Formative Assessment Responses in Sciencecited by 31
  10. 2024Task Facet Learning: A Structured Approach to Prompt Optimizationcited by 19
  11. 2024Grade Like a Human: Rethinking Automated Assessment with Large Language Modelscited by 16
  12. 2024Misconfidence-based Demonstration Selection for LLM In-Context Learningcited by 14
  13. 2018Automatic Short Answer Grading and Feedback Using Text Mining Methodscited by 11
  14. 2024Can Large Language Models Make the Grade? An Empirical Study Evaluating LLMs Ability to Mark Short Answer Questions in K-12 Educationcited by 11
  15. 2024Content Knowledge Identification with Multi-Agent Large Language Models (LLMs)cited by 6
  16. 2024Explainable Automatic Grading with Neural Additive Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 11.

16 papers.

Sort:

By area:

  1. 2026Agent-as-a-Judgecited by 17
  2. 2025LLM-based Automated Grading with Human-in-the-Loopcited by 13
  3. 2025RefGrader: Automated Grading of Mathematical Competition Proofs using Agentic Workflowscited by 4
  4. 2025Pensieve Grader: An AI-Powered, Ready-to-Use Platform for Effortless Handwritten STEM Gradingcited by 3
  5. 2026Evidence-Decision-Feedback: Theory-Driven Adaptive Scaffolding for LLM Agentscited by 3
  6. 2026Confusion-Aware Rubric Optimization for LLM-based Automated Gradingcited by 3
  7. 2026Don't Overthink It: Inter-Rollout Action Agreement as a Free Adaptive-Compute Signal for LLM Agentscited by 2
  8. 2025On the effectiveness of LLMs for automatic grading of open-ended questions in Spanishcited by 1
  9. 2025A LLM-Driven Multi-Agent Systems for Professional Development of Mathematics Teacherscited by 1
  10. 2026Implicit Grading Bias in Large Language Models: How Writing Style Affects Automated Assessment Across Math, Programming, and Essay Taskscited by 1
  11. 2026"**Important** You should give me full credits!": Exploring Prompt Injection Attacks on LLM-Based Automatic Grading Systemscited by 1
  12. 2026AI Meets Mathematics Education: A Case Study on Supporting an Instructor in a Large Mathematics Class with Context-Aware AIcited by 0
4 more
  1. 2026From Tool to Teammate: LLM Coding Agents as Collaborative Partners for Behavioral Labeling in Educational Dialogue Analysiscited by 0
  2. 2026Learnable Assessment Skills for LLM-based Automated Scoring: Rubric Construction via Iterative Optimizationcited by 0
  3. 2026A Theory-Guided LLM Pedagogical Agent for STEM+C Scaffolding Without Over-Reliancecited by 0
  4. 2026LLMs as Teaching Assistants for Mathematics Exam Grading: Reliability, and Practical Usabilitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Language Models are Few-Shot Graders
  2. 2025SteLLA: A Structured Grading System Using LLMs with RAG
  3. 2025Enhancing LLM-Based Short Answer Grading with Retrieval-Augmented Generation
  4. 2024Beyond Scores: A Modular RAG-Based System for Automatic Short Answer Scoring with Feedback
  5. 2025AutoSCORE: Enhancing Automated Scoring with Multi-Agent Large Language Models via Structured Component Recognition
  6. 2024Auditing an Automatic Grading Model with deep Reinforcement Learning
  7. 2025SAS-Bench: A Fine-Grained Benchmark for Evaluating Short Answer Scoring with Large Language Models
  8. 2024Investigating Automatic Scoring and Feedback using Large Language Models