# Modeling and Analyzing Scorer Preferences in Short-Answer Math Questions

arXiv: https://arxiv.org/abs/2306.00791 · HTML: https://mlsanity.com/paper/2306.00791/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 48 references that are in the map.
- First posted on arXiv 1 Jun 2023.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Mistreatment](https://mlsanity.com/paper/1610.08452/) (2016)
- [Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey](https://mlsanity.com/paper/1912.05170/) (2019)
- [MathBERT: A Pre-trained Language Model for General NLP Tasks in Mathematics Education](https://mlsanity.com/paper/2106.07340/) (2021)
- [Automatic Short Math Answer Grading via In-context Meta-learning](https://mlsanity.com/paper/2205.15219/) (2022)
- [Automated Scoring for Reading Comprehension via In-context BERT Tuning](https://mlsanity.com/paper/2205.09864/) (2022)
- [Mitigating Biases in Student Performance Prediction via Attention-Based Personalized Federated Learning](https://mlsanity.com/paper/2208.01182/) (2022)
- [Tree-Based Representation and Generation of Natural and Mathematical Language](https://mlsanity.com/paper/2302.07974/) (2023)
- [Should College Dropout Prediction Models Include Protected Attributes?](https://mlsanity.com/paper/2103.15237/) (2021)
- [Methods for Ordinal Peer Grading](https://mlsanity.com/paper/1404.3656/) (2014)

## Abstract (arXiv)

Automated scoring of student responses to open-ended questions, including short-answer questions, has great potential to scale to a large number of responses. Recent approaches for automated scoring rely on supervised learning, i.e., training classifiers or fine-tuning language models on a small number of responses with human-provided score labels. However, since scoring is a subjective process, these human scores are noisy and can be highly variable, depending on the scorer. In this paper, we investigate a collection of models that account for the individual preferences and tendencies of each human scorer in the automated scoring task. We apply these models to a short-answer math response dataset where each response is scored (often differently) by multiple different human scorers. We conduct quantitative experiments to show that our scorer models lead to improved automated scoring accuracy. We also conduct quantitative experiments and case studies to analyze the individual preferences and tendencies of scorers. We found that scorers can be grouped into several obvious clusters, with each cluster having distinct features, and analyzed them in detail.