# S-GRADES -- Studying Generalization of Student Response Assessments in Diverse Evaluative Settings

arXiv: https://arxiv.org/abs/2603.10233 · HTML: https://mlsanity.com/paper/2603.10233/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 32 references that are in the map.
- First posted on arXiv 10 Mar 2026.

## Builds on (most cited first, 10 of 10)

- [Human-AI Collaborative Essay Scoring: A Dual-Process Framework with LLMs](https://mlsanity.com/paper/2401.06431/) (2024)
- [Language models and Automated Essay Scoring](https://mlsanity.com/paper/1909.09482/) (2019)
- [Survey on Automated Short Answer Grading with Deep Learning: from Word Embeddings to Transformers](https://mlsanity.com/paper/2204.03503/) (2022)
- [Can AI grade your essays? A comparative analysis of large language models and teacher ratings in multidimensional essay scoring](https://mlsanity.com/paper/2411.16337/) (2024)
- [Automated Scoring for Reading Comprehension via In-context BERT Tuning](https://mlsanity.com/paper/2205.09864/) (2022)
- [Performance of the Pre-Trained Large Language Model GPT-4 on Automated Short Answer Grading](https://mlsanity.com/paper/2309.09338/) (2023)
- [Automated Essay Scoring Using Transformer Models](https://mlsanity.com/paper/2110.06874/) (2021)
- [Automated Long Answer Grading with RiceChem Dataset](https://mlsanity.com/paper/2404.14316/) (2024)
- [Can LLMs Grade Short-Answer Reading Comprehension Questions : An Empirical Study with a Novel Dataset](https://mlsanity.com/paper/2310.18373/) (2023)
- [Towards Scalable Automated Grading: Leveraging Large Language Models for Conceptual Question Evaluation in Engineering](https://mlsanity.com/paper/2411.03659/) (2024)

## Abstract (arXiv)

Evaluating student responses, from long essays to short factual answers, is a key challenge in educational NLP. Automated Essay Scoring (AES) focuses on holistic writing qualities such as coherence and argumentation, while Automatic Short Answer Grading (ASAG) emphasizes factual correctness and conceptual understanding. Despite their shared goal, these paradigms have progressed in isolation with fragmented datasets, inconsistent metrics, and separate communities. We introduce S-GRADES (Studying Generalization of Student Response Assessments in Diverse Evaluative Settings), a web-based benchmark that consolidates 14 diverse grading datasets under a unified interface with standardized access and reproducible evaluation protocols. The benchmark is fully open-source and designed for extensibility, enabling continuous integration of new datasets and evaluation settings. To demonstrate the utility of S-GRADES, we evaluate three state-of-the-art large language models across the benchmark using multiple reasoning strategies in prompting. We further examine the effects of exemplar selection and cross-dataset exemplar transfer. Our analyses illustrate how benchmark-driven evaluation reveals reliability and generalization gaps across essay and short-answer grading tasks, highlighting the importance of standardized, cross-paradigm assessment.