# Unveiling the Tapestry of Automated Essay Scoring: A Comprehensive Investigation of Accuracy, Fairness, and Generalizability

arXiv: https://arxiv.org/abs/2401.05655 · HTML: https://mlsanity.com/paper/2401.05655/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 42 references that are in the map.
- First posted on arXiv 11 Jan 2024.

## Builds on (most cited first, 8 of 8)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [Automatic Text Scoring Using Neural Networks](https://mlsanity.com/paper/1606.04289/) (2016)
- [Language models and Automated Essay Scoring](https://mlsanity.com/paper/1909.09482/) (2019)
- [Automated essay scoring with string kernels and word embeddings](https://mlsanity.com/paper/1804.07954/) (2018)
- [SkipFlow: Incorporating Neural Coherence Features for End-to-End Automatic Text Scoring](https://mlsanity.com/paper/1711.04981/) (2017)
- [Prompt Agnostic Essay Scorer: A Domain Generalization Approach to Cross-prompt Automated Essay Scoring](https://mlsanity.com/paper/2008.01441/) (2020)

## Built on by (most cited first, 8 of 8)

- [EssayJudge: A Multi-Granular Benchmark for Assessing Automated Essay Scoring Capabilities of Multimodal Large Language Models](https://mlsanity.com/paper/2502.11916/) (2025)
- [CAFES: A Collaborative Multi-Agent Framework for Multi-Granular Multimodal Essay Scoring](https://mlsanity.com/paper/2505.13965/) (2025)
- [How well can LLMs Grade Essays in Arabic?](https://mlsanity.com/paper/2501.16516/) (2025)
- [Does the Prompt-based Large Language Model Recognize Students' Demographics and Introduce Bias in Essay Scoring?](https://mlsanity.com/paper/2504.21330/) (2025)
- [Operationalizing Automated Essay Scoring: A Human-Aware Approach](https://mlsanity.com/paper/2506.21603/) (2025)
- [Mitigating Bias in Automated Essay Scoring for ESL Learners via Contrastive Learning](https://mlsanity.com/paper/2601.16724/) (2026)
- [The Order Matters: Sequential Fine-Tuning of LLaMA for Coherent Automated Essay Scoring](https://mlsanity.com/paper/2606.10327/) (2026)
- [Demographic Metadata as Construct-Irrelevant Noise in DistilBERT-Based Automated Essay Scoring](https://mlsanity.com/paper/2606.21066/) (2026)

## Abstract (arXiv)

Automatic Essay Scoring (AES) is a well-established educational pursuit that employs machine learning to evaluate student-authored essays. While much effort has been made in this area, current research primarily focuses on either (i) boosting the predictive accuracy of an AES model for a specific prompt (i.e., developing prompt-specific models), which often heavily relies on the use of the labeled data from the same target prompt; or (ii) assessing the applicability of AES models developed on non-target prompts to the intended target prompt (i.e., developing the AES models in a cross-prompt setting). Given the inherent bias in machine learning and its potential impact on marginalized groups, it is imperative to investigate whether such bias exists in current AES methods and, if identified, how it intervenes with an AES model's accuracy and generalizability. Thus, our study aimed to uncover the intricate relationship between an AES model's accuracy, fairness, and generalizability, contributing practical insights for developing effective AES models in real-world education. To this end, we meticulously selected nine prominent AES methods and evaluated their performance using seven metrics on an open-sourced dataset, which contains over 25,000 essays and various demographic information about students such as gender, English language learner status, and economic status. Through extensive evaluations, we demonstrated that: (1) prompt-specific models tend to outperform their cross-prompt counterparts in terms of predictive accuracy; (2) prompt-specific models frequently exhibit a greater bias towards students of different economic statuses compared to cross-prompt models; (3) in the pursuit of generalizability, traditional machine learning models coupled with carefully engineered features hold greater potential for achieving both high accuracy and fairness than complex neural network models.