# Enhancing Marker Scoring Accuracy through Ordinal Confidence Modelling in Educational Assessments

arXiv: https://arxiv.org/abs/2505.23315 · HTML: https://mlsanity.com/paper/2505.23315/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 29 references that are in the map.
- First posted on arXiv 29 May 2025.

## Builds on (most cited first, 4 of 4)

- [An Effectiveness Metric for Ordinal Classification: Formal Properties and Experimental Results](https://mlsanity.com/paper/2006.01245/) (2020)
- [Using Sampling to Estimate and Improve Performance of Automated Scoring Systems with Guarantees](https://mlsanity.com/paper/2111.08906/) (2021)
- [Normative Challenges of Risk Regulation of Artificial Intelligence and Automated Decision-Making](https://mlsanity.com/paper/2211.06203/) (2022)
- [Using Active Learning Methods to Strategically Select Essays for Automated Scoring](https://mlsanity.com/paper/2301.00628/) (2023)

## Built on by (most cited first, 4 of 4)

- [Beyond Holistic Scores: Automatic Trait-Based Quality Scoring of Argumentative Essays](https://mlsanity.com/paper/2602.04604/) (2026)
- [Architectural Insights for Post-Tornado Damage Recognition](https://mlsanity.com/paper/2602.14523/) (2026)
- [Incentives Of EdTech: A Systematic Review Of EduNLP Research](https://mlsanity.com/paper/2606.13691/) (2026)
- [Controlling Implicit Shortcut Reliance in L2 Spoken English Auto-markers](https://mlsanity.com/paper/2607.16085/) (2026)

## Abstract (arXiv)

A key ethical challenge in Automated Essay Scoring (AES) is ensuring that scores are only released when they meet high reliability standards. Confidence modelling addresses this by assigning a reliability estimate measure, in the form of a confidence score, to each automated score. In this study, we frame confidence estimation as a classification task: predicting whether an AES-generated score correctly places a candidate in the appropriate CEFR level. While this is a binary decision, we leverage the inherent granularity of the scoring domain in two ways. First, we reformulate the task as an n-ary classification problem using score binning. Second, we introduce a set of novel Kernel Weighted Ordinal Categorical Cross Entropy (KWOCCE) loss functions that incorporate the ordinal structure of CEFR labels. Our best-performing model achieves an F1 score of 0.97, and enables the system to release 47% of scores with 100% CEFR agreement and 99% with at least 95% CEFR agreement -compared to approximately 92% (approx.) CEFR agreement from the standalone AES model where we release all AM predicted scores.