# Automated Long Answer Grading with RiceChem Dataset arXiv: https://arxiv.org/abs/2404.14316 ยท HTML: https://mlsanity.com/paper/2404.14316/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Builds on 17 of its 32 references that are in the map. - First posted on arXiv 22 Apr 2024. ## Builds on (most cited first, 17 of 17) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019) - [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023) - [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023) - [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017) - [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023) - [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024) - [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024) - [Textbooks Are All You Need II: phi-1.5 technical report](https://mlsanity.com/paper/2309.05463/) (2023) - [On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines](https://mlsanity.com/paper/2006.04884/) (2020) - [CLASS: A Design Framework for building Intelligent Tutoring Systems based on Learning Science principles](https://mlsanity.com/paper/2305.13272/) (2023) - [Many Hands Make Light Work: Using Essay Traits to Automatically Score Essays](https://mlsanity.com/paper/2102.00781/) (2021) - [Pedagogical Alignment of Large Language Models](https://mlsanity.com/paper/2402.05000/) (2024) - [Performance of the Pre-Trained Large Language Model GPT-4 on Automated Short Answer Grading](https://mlsanity.com/paper/2309.09338/) (2023) - [Code Soliloquies for Accurate Calculations in Large Language Models](https://mlsanity.com/paper/2309.12161/) (2023) ## Built on by (most cited first, 10 of 10) - [Student Data Paradox and Curious Case of Single Student-Tutor Model: Regressive Side Effects of Training LLMs for Personalized Learning](https://mlsanity.com/paper/2404.15156/) (2024) - [Breaking the Chain: A Causal Analysis of LLM Faithfulness to Intermediate Structures](https://mlsanity.com/paper/2603.16475/) (2026) - [Decision Models vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places](https://mlsanity.com/paper/2609.29769/) (2026) - [Concept-based Rubrics Improve LLM Formative Assessment and Data Synthesis](https://mlsanity.com/paper/2504.03877/) (2025) - [SAS-Bench: A Fine-Grained Benchmark for Evaluating Short Answer Scoring with Large Language Models](https://mlsanity.com/paper/2505.07247/) (2025) - [Ratas framework: A comprehensive genai-based approach to rubric-based marking of real-world textual exams](https://mlsanity.com/paper/2505.23818/) (2025) - [CLEAR-3K: Assessing Causal Explanatory Capabilities in Language Models](https://mlsanity.com/paper/2506.17180/) (2025) - [When Can We Trust LLM Graders? Calibrating Confidence for Automated Assessment](https://mlsanity.com/paper/2603.29559/) (2026) - [S-GRADES -- Studying Generalization of Student Response Assessments in Diverse Evaluative Settings](https://mlsanity.com/paper/2603.10233/) (2026) - [A Human-in-the-Loop Framework for AI-Assisted Scoring in Large-Scale Writing Assessment](https://mlsanity.com/paper/2609.05143/) (2026) ## Abstract (arXiv) We introduce a new area of study in the field of educational Natural Language Processing: Automated Long Answer Grading (ALAG). Distinguishing itself from Automated Short Answer Grading (ASAG) and Automated Essay Grading (AEG), ALAG presents unique challenges due to the complexity and multifaceted nature of fact-based long answers. To study ALAG, we introduce RiceChem, a dataset derived from a college chemistry course, featuring real student responses to long-answer questions with an average word count notably higher than typical ASAG datasets. We propose a novel approach to ALAG by formulating it as a rubric entailment problem, employing natural language inference models to verify whether each criterion, represented by a rubric item, is addressed in the student's response. This formulation enables the effective use of MNLI for transfer learning, significantly improving the performance of models on the RiceChem dataset. We demonstrate the importance of rubric-based formulation in ALAG, showcasing its superiority over traditional score-based approaches in capturing the nuances of student responses. We also investigate the performance of models in cold start scenarios, providing valuable insights into the practical deployment considerations in educational settings. Lastly, we benchmark state-of-the-art open-sourced Large Language Models (LLMs) on RiceChem and compare their results to GPT models, highlighting the increased complexity of ALAG compared to ASAG. Despite leveraging the benefits of a rubric-based approach and transfer learning from MNLI, the lower performance of LLMs on RiceChem underscores the significant difficulty posed by the ALAG task. With this work, we offer a fresh perspective on grading long, fact-based answers and introduce a new dataset to stimulate further research in this important area. Code: urlhttps://github.com/luffycodes/Automated-Long-Answer-Grading.