# Language models and Automated Essay Scoring

arXiv: https://arxiv.org/abs/1909.09482 · HTML: https://mlsanity.com/paper/1909.09482/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Sep 2019.

## Built on by (most cited first, 30 of 35)

- [Applying Large Language Models and Chain-of-Thought for Automatic Scoring](https://mlsanity.com/paper/2312.03748/) (2023)
- [Fine-tuning ChatGPT for Automatic Scoring](https://mlsanity.com/paper/2310.10072/) (2023)
- [On the Use of BERT for Automated Essay Scoring: Joint Learning of Multi-Scale Essay Representation](https://mlsanity.com/paper/2205.03835/) (2022)
- [Human-AI Collaborative Essay Scoring: A Dual-Process Framework with LLMs](https://mlsanity.com/paper/2401.06431/) (2024)
- [Can GPT-4 do L2 analytic assessment?](https://mlsanity.com/paper/2404.18557/) (2024)
- [Automated Essay Scoring Using Transformer Models](https://mlsanity.com/paper/2110.06874/) (2021)
- [When Automated Assessment Meets Automated Content Generation: Examining Text Quality in the Era of GPTs](https://mlsanity.com/paper/2309.14488/) (2023)
- [Automated essay scoring using efficient transformer-based language models](https://mlsanity.com/paper/2102.13136/) (2021)
- [Automated Text Scoring in the Age of Generative AI for the GPU-poor](https://mlsanity.com/paper/2407.01873/) (2024)
- [LLMs can Perform Multi-Dimensional Analytic Writing Assessments: A Case Study of L2 Graduate-Level Academic English Writing](https://mlsanity.com/paper/2502.11368/) (2025)
- [Short-answer scoring with ensembles of pretrained language models](https://mlsanity.com/paper/2202.11558/) (2022)
- [Unveiling the Tapestry of Automated Essay Scoring: A Comprehensive Investigation of Accuracy, Fairness, and Generalizability](https://mlsanity.com/paper/2401.05655/) (2024)
- [Autoregressive Score Generation for Multi-trait Essay Scoring](https://mlsanity.com/paper/2403.08332/) (2024)
- [Beyond Agreement: Diagnosing the Rationale Alignment of Automated Essay Scoring Methods based on Linguistically-informed Counterfactuals](https://mlsanity.com/paper/2405.19433/) (2024)
- [FOKE: A Personalized and Explainable Education Framework Integrating Foundation Models, Knowledge Graphs, and Prompt Engineering](https://mlsanity.com/paper/2405.03734/) (2024)
- [Does the Prompt-based Large Language Model Recognize Students' Demographics and Introduce Bias in Essay Scoring?](https://mlsanity.com/paper/2504.21330/) (2025)
- [Opportunities and Challenges of LLMs in Education: An NLP Perspective](https://mlsanity.com/paper/2507.22753/) (2025)
- [Argumentation Element Annotation Modeling using XLNet](https://mlsanity.com/paper/2311.06239/) (2023)
- [H-AES: Towards Automated Essay Scoring for Hindi](https://mlsanity.com/paper/2302.14635/) (2023)
- [On the Suitability of pre-trained foundational LLMs for Analysis in German Legal Education](https://mlsanity.com/paper/2412.15902/) (2024)
- [The effects of data size on Automated Essay Scoring engines](https://mlsanity.com/paper/2108.13275/) (2021)
- [Toward Educator-focused Automated Scoring Systems for Reading and Writing](https://mlsanity.com/paper/2112.11973/) (2021)
- [Tapping the Potential of Coherence and Syntactic Features in Neural Models for Automatic Essay Scoring](https://mlsanity.com/paper/2211.13373/) (2022)
- [Comparison Study Between Token Classification and Sequence Classification In Text Classification](https://mlsanity.com/paper/2211.13899/) (2022)
- [Comparing Human and AI Rater Effects Using the Many-Facet Rasch Model](https://mlsanity.com/paper/2505.18486/) (2025)
- [Long Context Automated Essay Scoring with Language Models](https://mlsanity.com/paper/2509.10417/) (2025)
- [Comparison of Scoring Rationales Between Large Language Models and Human Raters](https://mlsanity.com/paper/2509.23412/) (2025)
- [Reconstructing Item Characteristic Curves using Fine-Tuned Large Language Models](https://mlsanity.com/paper/2601.02580/) (2026)
- [Automated Essay Scoring Incorporating Annotations from Automated Feedback Systems](https://mlsanity.com/paper/2505.22771/) (2025)
- [PeerCoPilot: A Language Model-Powered Assistant for Behavioral Health Organizations](https://mlsanity.com/paper/2511.21721/) (2025)

## Abstract (arXiv)

In this paper, we present a new comparative study on automatic essay scoring (AES). The current state-of-the-art natural language processing (NLP) neural network architectures are used in this work to achieve above human-level accuracy on the publicly available Kaggle AES dataset. We compare two powerful language models, BERT and XLNet, and describe all the layers and network architectures in these models. We elucidate the network architectures of BERT and XLNet using clear notation and diagrams and explain the advantages of transformer architectures over traditional recurrent neural network architectures. Linear algebra notation is used to clarify the functions of transformers and attention mechanisms. We compare the results with more traditional methods, such as bag of words (BOW) and long short term memory (LSTM) networks.