# Automated Essay Scoring Using Transformer Models

arXiv: https://arxiv.org/abs/2110.06874 · HTML: https://mlsanity.com/paper/2110.06874/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 95 references that are in the map.
- First posted on arXiv 13 Oct 2021.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation](https://mlsanity.com/paper/2010.16061/) (2020)
- [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](https://mlsanity.com/paper/2012.06678/) (2020)
- [UniT: Multimodal Multitask Learning with a Unified Transformer](https://mlsanity.com/paper/2102.10772/) (2021)
- [German's Next Language Model](https://mlsanity.com/paper/2010.10906/) (2020)
- [Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case](https://mlsanity.com/paper/2001.08317/) (2020)
- [The Cost of Training NLP Models: A Concise Overview](https://mlsanity.com/paper/2004.08900/) (2020)
- [Zero-shot Text Classification With Generative Language Models](https://mlsanity.com/paper/1912.10165/) (2019)
- [Measuring the Algorithmic Efficiency of Neural Networks](https://mlsanity.com/paper/2005.04305/) (2020)
- [Automatic Text Scoring Using Neural Networks](https://mlsanity.com/paper/1606.04289/) (2016)
- [Language models and Automated Essay Scoring](https://mlsanity.com/paper/1909.09482/) (2019)
- [Machine Learning Meets Natural Language Processing -- The story so far](https://mlsanity.com/paper/2104.10213/) (2021)

## Built on by (most cited first, 14 of 14)

- [Contextualizing Argument Quality Assessment with Relevant Knowledge](https://mlsanity.com/paper/2305.12280/) (2023)
- [Data Augmentation for Automated Essay Scoring using Transformer Models](https://mlsanity.com/paper/2210.12809/) (2022)
- [H-AES: Towards Automated Essay Scoring for Hindi](https://mlsanity.com/paper/2302.14635/) (2023)
- [G-SciEdBERT: A Contextualized LLM for Science Assessment Tasks in German](https://mlsanity.com/paper/2402.06584/) (2024)
- [Transformer-based Joint Modelling for Automatic Essay Scoring and Off-Topic Detection](https://mlsanity.com/paper/2404.08655/) (2024)
- [The Effectiveness of a Dynamic Loss Function in Neural Network Based Automated Essay Scoring](https://mlsanity.com/paper/2305.10447/) (2023)
- [Comparative Study and Framework for Automated Summariser Evaluation: LangChain and Hybrid Algorithms](https://mlsanity.com/paper/2310.02759/) (2023)
- [Comparing Human and AI Rater Effects Using the Many-Facet Rasch Model](https://mlsanity.com/paper/2505.18486/) (2025)
- [Comparison of Scoring Rationales Between Large Language Models and Human Raters](https://mlsanity.com/paper/2509.23412/) (2025)
- [Quantifying Holistic Review: A Multi-Modal Approach to College Admissions Prediction](https://mlsanity.com/paper/2507.15862/) (2025)
- [IELTS Writing Revision Platform with Automated Essay Scoring and Adaptive Feedback](https://mlsanity.com/paper/2512.24460/) (2025)
- [Beyond Pipelines: A Fundamental Study on the Rise of Generative-Retrieval Architectures in Web Research](https://mlsanity.com/paper/2602.17450/) (2026)
- [Evaluating Austrian A-Level German Essays with Large Language Models for Automated Essay Scoring](https://mlsanity.com/paper/2603.06066/) (2026)
- [S-GRADES -- Studying Generalization of Student Response Assessments in Diverse Evaluative Settings](https://mlsanity.com/paper/2603.10233/) (2026)

## Abstract (arXiv)

Automated essay scoring (AES) is gaining increasing attention in the education sector as it significantly reduces the burden of manual scoring and allows ad hoc feedback for learners. Natural language processing based on machine learning has been shown to be particularly suitable for text classification and AES. While many machine-learning approaches for AES still rely on a bag-of-words (BOW) approach, we consider a transformer-based approach in this paper, compare its performance to a logistic regression model based on the BOW approach and discuss their differences. The analysis is based on 2,088 email responses to a problem-solving task, that were manually labeled in terms of politeness. Both transformer models considered in that analysis outperformed without any hyper-parameter tuning the regression-based model. We argue that for AES tasks such as politeness classification, the transformer-based approach has significant advantages, while a BOW approach suffers from not taking word order into account and reducing the words to their stem. Further, we show how such models can help increase the accuracy of human raters, and we provide a detailed instruction on how to implement transformer-based models for one's own purpose.