mlsanityData to 7 Oct 2026

CHiL(L)Grader: Calibrated Human-in-the-Loop Short-Answer Grading

arXiv 2603.11957 ↗First posted 12 Mar 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Scaling educational assessment with large language models requires not just accuracy, but the ability to recognize when predictions are trustworthy. Instruction-tuned models tend to be overconfident, and their reliability deteriorates as curricula evolve, making fully autonomous deployment unsafe in high-stakes settings. We introduce CHiL(L)Grader, the first automated grading framework that incorporates calibrated confidence estimation into a human-in-the-loop workflow. Using post-hoc temperature scaling, confidence-based selective prediction, and continual learning, CHiL(L)Grader automates only high-confidence predictions while routing uncertain cases to human graders, and adapts to evolving rubrics and unseen questions. Across three short-answer grading datasets, CHiL(L)Grader automatically scores 35-65% of responses at expert-level quality (QWK >= 0.80). A QWK gap of 0.347 between accepted and rejected predictions confirms the effectiveness of the confidence-based routing. Each correction cycle strengthens the model's grading capability as it learns from teacher feedback. These results show that uncertainty quantification is key for reliable AI-assisted grading.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 20 of its 36 references that are in the map. First posted on arXiv 12 Mar 2026.

Builds on

20 of 36 references.

References · 20

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2023Mistral 7Bcited by 3,065
  5. 2024Qwen2 Technical Reportcited by 2,076
  6. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  7. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  8. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  9. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  10. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  11. 2019Measuring Calibration in Deep Learningcited by 443
  12. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 296
8 more
  1. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253
  2. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 138
  3. 2024Calibrating Language Models with Adaptive Temperature Scalingcited by 57
  4. 2021Estimating Expected Calibration Errorscited by 24
  5. 2024Using Large Language Models for Automated Grading of Student Writing about Sciencecited by 12
  6. 2025Enhancing LLM-Based Short Answer Grading with Retrieval-Augmented Generationcited by 7
  7. 2024Reinforcement learning for question answering in programming domain using public community scoring as a human feedbackcited by 6
  8. 2025Advancing Student Writing Through Automated Syntax Feedbackcited by 1

Built on by

Not cited yet.