mlsanityData to 7 Oct 2026

SteLLA: A Structured Grading System Using LLMs with RAG

arXiv 2501.09092 ↗First posted 15 Jan 2025cs.CL cs.AI cs.CYcited by 3

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have shown strong general capabilities in many applications. However, how to make them reliable tools for some specific tasks such as automated short answer grading (ASAG) remains a challenge. We present SteLLA (Structured Grading System Using LLMs with RAG) in which a) Retrieval Augmented Generation (RAG) approach is used to empower LLMs specifically on the ASAG task by extracting structured information from the highly relevant and reliable external knowledge based on the instructor-provided reference answer and rubric, b) an LLM performs a structured and question-answering-based evaluation of student answers to provide analytical grades and feedback. A real-world dataset that contains students' answers in an exam was collected from a college-level Biology course. Experiments show that our proposed system can achieve substantial agreement with the human grader while providing break-down grades and feedback on all the knowledge points examined in the problem. A qualitative and error analysis of the feedback generated by GPT4 shows that GPT4 is good at capturing facts while may be prone to inferring too much implication from the given text in the grading task which provides insights into the usage of LLMs in the ASAG system.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 28 of its 45 references that are in the map. First posted on arXiv 15 Jan 2025.

Builds on

28 of 45 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  8. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  9. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  10. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  11. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  12. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
16 more
  1. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  2. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 241
  3. 2021The Curious Case of Hallucinations in Neural Machine Translationcited by 161
  4. 2020Factual Error Correction for Abstractive Summarization Modelscited by 108
  5. 2021Efficient Nearest Neighbor Language Modelscited by 93
  6. 2019Answers Unite! Unsupervised Metrics for Reinforced Summarization Modelscited by 92
  7. 2019Question Answering as an Automatic Evaluation Metric for News Article Summarizationcited by 63
  8. 2022On the Use of BERT for Automated Essay Scoring: Joint Learning of Multi-Scale Essay Representationcited by 46
  9. 2023Generating High-Precision Feedback for Programming Syntax Errors using Large Language Modelscited by 34
  10. 2024Can Large Language Models Automatically Score Proficiency of Written Essays?cited by 34
  11. 2021Data-QuestEval: A Referenceless Metric for Data-to-Text Semantic Evaluationcited by 21
  12. 2023Towards LLM-based Autograding for Short Textual Answerscited by 20
  13. 2017A Semantic QA-Based Approach for Text Summarization Evaluationcited by 17
  14. 2021Rethinking Automatic Evaluation in Sentence Simplificationcited by 17
  15. 2018Automatic Short Answer Grading and Feedback Using Text Mining Methodscited by 11
  16. 2023Short Answer Grading Using One-shot Prompting and Text Similarity Scoring Modelcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Rubric-Conditioned LLM Grading: Alignment, Uncertainty, and Robustnesscited by 4
  2. 2025Pensieve Grader: An AI-Powered, Ready-to-Use Platform for Effortless Handwritten STEM Gradingcited by 3
  3. 2026GRASP: Graph-Retrieval Automated Scoring Pipeline for Label-Free Multi-Topic Essay Gradingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing LLM-Based Short Answer Grading with Retrieval-Augmented Generation
  2. 2024Grade Like a Human: Rethinking Automated Assessment with Large Language Models
  3. 2024Beyond Scores: A Modular RAG-Based System for Automatic Short Answer Scoring with Feedback
  4. 2025Language Models are Few-Shot Graders
  5. 2025Focusing on Students, not Machines: Grounded Question Generation and Automated Answer Grading
  6. 2024Evaluating Students' Open-ended Written Responses with LLMs: Using the RAG Framework for GPT-3.5, GPT-4, Claude-3, and Mistral-Large
  7. 2024A LLM-Powered Automatic Grading Framework with Human-Level Guidelines Optimization
  8. 2025Aligning LLMs for the Classroom with Knowledge-Based Retrieval -- A Comparative RAG Study