# Using Sampling to Estimate and Improve Performance of Automated Scoring Systems with Guarantees

arXiv: https://arxiv.org/abs/2111.08906 · HTML: https://mlsanity.com/paper/2111.08906/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 45 references that are in the map.
- First posted on arXiv 17 Nov 2021.

## Builds on (most cited first, 12 of 12)

- [Unifying Human and Statistical Evaluation for Natural Language Generation](https://mlsanity.com/paper/1904.02792/) (2019)
- [The price of debiasing automatic metrics in natural language evaluation](https://mlsanity.com/paper/1807.02202/) (2018)
- [What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structure](https://mlsanity.com/paper/2101.00387/) (2021)
- [Get It Scored Using AutoSAS -- An Automated System for Scoring Short Answers](https://mlsanity.com/paper/2012.11243/) (2020)
- [The statistical advantage of automatic NLG metrics at the system level](https://mlsanity.com/paper/2105.12437/) (2021)
- [Approximate Selection with Guarantees using Proxies](https://mlsanity.com/paper/2004.00827/) (2020)
- [Automated Essay Scoring based on Two-Stage Learning](https://mlsanity.com/paper/1901.07744/) (2019)
- [Speaker-Conditioned Hierarchical Modeling for Automated Speech Scoring](https://mlsanity.com/paper/2109.00928/) (2021)
- [My Teacher Thinks The World Is Flat! Interpreting Automatic Essay Scoring Mechanism](https://mlsanity.com/paper/2012.13872/) (2020)
- [Multi-modal Automated Speech Scoring using Attention Fusion](https://mlsanity.com/paper/2005.08182/) (2020)
- [Towards Modelling Coherence in Spoken Discourse](https://mlsanity.com/paper/2101.00056/) (2020)
- [AES Systems Are Both Overstable And Oversensitive: Explaining Why And Proposing Defenses](https://mlsanity.com/paper/2109.11728/) (2021)

## Built on by (most cited first, 5 of 5)

- [A Chain-of-Thought Prompting Approach with LLMs for Evaluating Students' Formative Assessment Responses in Science](https://mlsanity.com/paper/2403.14565/) (2024)
- [Toward More Effective Human Evaluation for Machine Translation](https://mlsanity.com/paper/2204.05307/) (2022)
- [Automated Speech Scoring System Under The Lens: Evaluating and interpreting the linguistic cues for language proficiency](https://mlsanity.com/paper/2111.15156/) (2021)
- [Enhancing Marker Scoring Accuracy through Ordinal Confidence Modelling in Educational Assessments](https://mlsanity.com/paper/2505.23315/) (2025)
- [H-AES: Towards Automated Essay Scoring for Hindi](https://mlsanity.com/paper/2302.14635/) (2023)

## Abstract (arXiv)

Automated Scoring (AS), the natural language processing task of scoring essays and speeches in an educational testing setting, is growing in popularity and being deployed across contexts from government examinations to companies providing language proficiency services. However, existing systems either forgo human raters entirely, thus harming the reliability of the test, or score every response by both human and machine thereby increasing costs. We target the spectrum of possible solutions in between, making use of both humans and machines to provide a higher quality test while keeping costs reasonable to democratize access to AS. In this work, we propose a combination of the existing paradigms, sampling responses to be scored by humans intelligently. We propose reward sampling and observe significant gains in accuracy (19.80% increase on average) and quadratic weighted kappa (QWK) (25.60% on average) with a relatively small human budget (30% samples) using our proposed sampling. The accuracy increase observed using standard random and importance sampling baselines are 8.6% and 12.2% respectively. Furthermore, we demonstrate the system's model agnostic nature by measuring its performance on a variety of models currently deployed in an AS setting as well as pseudo models. Finally, we propose an algorithm to estimate the accuracy/QWK with statistical guarantees (Our code is available at https://git.io/J1IOy).