# Impacts of Histories and Models on LLM Grading: A Study in Advanced Software Engineering Courses

arXiv: https://arxiv.org/abs/2606.08400 · HTML: https://mlsanity.com/paper/2606.08400/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 16 references that are in the map.
- First posted on arXiv 7 Jun 2026.

## Builds on (most cited first, 5 of 5)

- [Exploring LLM Prompting Strategies for Joint Essay Scoring and Feedback Generation](https://mlsanity.com/paper/2404.15845/) (2024)
- [Right Answer, Wrong Score: Uncovering the Inconsistencies of LLM Evaluation in Multiple-Choice Question Answering](https://mlsanity.com/paper/2503.14996/) (2025)
- [Efficient Solutions For An Intriguing Failure of LLMs: Long Context Window Does Not Mean LLMs Can Analyze Long Sequences Flawlessly](https://mlsanity.com/paper/2408.01866/) (2024)
- [Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settings](https://mlsanity.com/paper/2503.15620/) (2025)
- [CoGrader: Transforming Instructors' Assessment of Project Reports through Collaborative LLM Integration](https://mlsanity.com/paper/2507.20655/) (2025)

## Abstract (arXiv)

Graduate-level research reading report assessment creates a substantial labor burden for educators. While large language models (LLMs) hold great potential for automating academic grading, their reliability for this specialized task remains understudied, particularly regarding grading consistency, the lack of which represents a primary obstacle to educational fairness. This paper proposes a human-aligned LLM-assisted grading workflow and presents a case study based on 180 student submissions from a graduate advanced software engineering course. We evaluate two mainstream LLMs, Grok and GPT, in terms of grading consistency and alignment with human scores. We find LLMs exhibit distinct levels of intra-model consistency and significant inter-model grading inconsistencies, while simple ensemble approaches cannot improve alignment with human evaluation. Critically, continuous interaction history drives systematic drift in models' grading standards away from human expert scores. Our findings demonstrate LLMs' potential in reducing grading workload for educators in graduate education, while highlighting that indiscriminate LLM grading may introduce systemic unfairness, suggesting that specific operational practices are required to mitigate such disparities.