# Using ChatGPT to Score Essays and Short-Form Constructed Responses

arXiv: https://arxiv.org/abs/2408.09540 · HTML: https://mlsanity.com/paper/2408.09540/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 27 references that are in the map.
- First posted on arXiv 18 Aug 2024.

## Builds on (most cited first, 5 of 5)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [A Survey of Human-in-the-loop for Machine Learning](https://mlsanity.com/paper/2108.00941/) (2021)
- [Cultural Bias and Cultural Alignment of Large Language Models](https://mlsanity.com/paper/2311.14096/) (2023)
- [Fine-tuning ChatGPT for Automatic Scoring](https://mlsanity.com/paper/2310.10072/) (2023)
- [Identifying and Improving Disability Bias in GPT-Based Resume Screening](https://mlsanity.com/paper/2402.01732/) (2024)

## Built on by (most cited first, 2 of 2)

- [Agreement Between Large Language Models and Human Raters in Essay Scoring: A Research Synthesis](https://mlsanity.com/paper/2512.14561/) (2025)
- [Exploring the Utilities of the Rationales from Large Language Models to Enhance Automated Essay Scoring](https://mlsanity.com/paper/2510.27131/) (2025)

## Abstract (arXiv)

This study aimed to determine if ChatGPT's large language models could match the scoring accuracy of human and machine scores from the ASAP competition. The investigation focused on various prediction models, including linear regression, random forest, gradient boost, and boost. ChatGPT's performance was evaluated against human raters using quadratic weighted kappa (QWK) metrics. Results indicated that while ChatGPT's gradient boost model achieved QWKs close to human raters for some data sets, its overall performance was inconsistent and often lower than human scores. The study highlighted the need for further refinement, particularly in handling biases and ensuring scoring fairness. Despite these challenges, ChatGPT demonstrated potential for scoring efficiency, especially with domain-specific fine-tuning. The study concludes that ChatGPT can complement human scoring but requires additional development to be reliable for high-stakes assessments. Future research should improve model accuracy, address ethical considerations, and explore hybrid models combining ChatGPT with empirical methods.