# Beyond Scores: A Modular RAG-Based System for Automatic Short Answer Scoring with Feedback

arXiv: https://arxiv.org/abs/2409.20042 · HTML: https://mlsanity.com/paper/2409.20042/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 47 references that are in the map.
- First posted on arXiv 30 Sep 2024.

## Builds on (most cited first, 17 of 17)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs](https://mlsanity.com/paper/2406.11695/) (2024)
- [Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertainty](https://mlsanity.com/paper/2401.06730/) (2024)
- [Towards Explainable NLP: A Generative Explanation Framework for Text Classification](https://mlsanity.com/paper/1811.00196/) (2018)
- [Towards LLM-based Autograding for Short Textual Answers](https://mlsanity.com/paper/2309.11508/) (2023)
- [Get It Scored Using AutoSAS -- An Automated System for Scoring Short Answers](https://mlsanity.com/paper/2012.11243/) (2020)
- [Distilling ChatGPT for Explainable Automated Student Answer Assessment](https://mlsanity.com/paper/2305.12962/) (2023)
- [Investigating Automatic Scoring and Feedback using Large Language Models](https://mlsanity.com/paper/2405.00602/) (2024)
- [Graded Relevance Scoring of Written Essays with Dense Retrieval](https://mlsanity.com/paper/2405.05200/) (2024)

## Built on by (most cited first, 5 of 5)

- [Personalizing Student-Agent Interactions Using Log-Contextualized Retrieval-Augmented Generation (RAG)](https://mlsanity.com/paper/2505.17238/) (2025)
- [Can MLLMs generate human-like feedback in grading multimodal short answers?](https://mlsanity.com/paper/2412.19755/) (2024)
- [Ensemble ToT of LLMs and Its Application to Automatic Grading System for Supporting Self-Learning](https://mlsanity.com/paper/2502.16399/) (2025)
- [Persona Matters: Effects of Activation Steering on Short Answer Generation and Scoring](https://mlsanity.com/paper/2604.07102/) (2026)
- [Automatically Inferring Teachers' Geometric Content Knowledge: A Skills Based Approach](https://mlsanity.com/paper/2604.13666/) (2026)

## Abstract (arXiv)

Automatic short answer scoring (ASAS) helps reduce the grading burden on educators but often lacks detailed, explainable feedback. Existing methods in ASAS with feedback (ASAS-F) rely on fine-tuning language models with limited datasets, which is resource-intensive and struggles to generalize across contexts. Recent approaches using large language models (LLMs) have focused on scoring without extensive fine-tuning. However, they often rely heavily on prompt engineering and either fail to generate elaborated feedback or do not adequately evaluate it. In this paper, we propose a modular retrieval augmented generation based ASAS-F system that scores answers and generates feedback in strict zero-shot and few-shot learning scenarios. We design our system to be adaptable to various educational tasks without extensive prompt engineering using an automatic prompt generation framework. Results show an improvement in scoring accuracy by 9% on unseen questions compared to fine-tuning, offering a scalable and cost-effective solution.