# On the Use of BERT for Automated Essay Scoring: Joint Learning of Multi-Scale Essay Representation

arXiv: https://arxiv.org/abs/2205.03835 · HTML: https://mlsanity.com/paper/2205.03835/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 34 references that are in the map.
- First posted on arXiv 8 May 2022.

## Builds on (most cited first, 11 of 11)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [R-Drop: Regularized Dropout for Neural Networks](https://mlsanity.com/paper/2106.14448/) (2021)
- [Automatic Text Scoring Using Neural Networks](https://mlsanity.com/paper/1606.04289/) (2016)
- [Language models and Automated Essay Scoring](https://mlsanity.com/paper/1909.09482/) (2019)
- [Automated essay scoring with string kernels and word embeddings](https://mlsanity.com/paper/1804.07954/) (2018)
- [SkipFlow: Incorporating Neural Coherence Features for End-to-End Automatic Text Scoring](https://mlsanity.com/paper/1711.04981/) (2017)
- [Phenotyping of Clinical Notes with Improved Document Classification Models Using Contextualized Neural Language Models](https://mlsanity.com/paper/1910.13664/) (2019)
- [Neural Automated Essay Scoring and Coherence Modeling for Adversarially Crafted Input](https://mlsanity.com/paper/1804.06898/) (2018)
- [Happy Are Those Who Grade without Seeing: A Multi-Task Learning Approach to Grade Essays Using Gaze Behaviour](https://mlsanity.com/paper/2005.12078/) (2020)

## Built on by (most cited first, 30 of 46)

- [Human-AI Collaborative Essay Scoring: A Dual-Process Framework with LLMs](https://mlsanity.com/paper/2401.06431/) (2024)
- [Can Large Language Models Automatically Score Proficiency of Written Essays?](https://mlsanity.com/paper/2403.06149/) (2024)
- [Prompt- and Trait Relation-aware Cross-prompt Essay Trait Scoring](https://mlsanity.com/paper/2305.16826/) (2023)
- [Rationale Behind Essay Scores: Enhancing S-LLM's Multi-Trait Essay Scoring with Rationale Generated by LLMs](https://mlsanity.com/paper/2410.14202/) (2024)
- [ChineseWebText: Large-scale High-quality Chinese Web Text Extracted with Effective Evaluation Model](https://mlsanity.com/paper/2311.01149/) (2023)
- [Autoregressive Multi-trait Essay Scoring via Reinforcement Learning with Scoring-aware Multiple Rewards](https://mlsanity.com/paper/2409.17472/) (2024)
- [Hidding the Ghostwriters: An Adversarial Evaluation of AI-Generated Student Essay Detection](https://mlsanity.com/paper/2402.00412/) (2024)
- [Autoregressive Score Generation for Multi-trait Essay Scoring](https://mlsanity.com/paper/2403.08332/) (2024)
- [Beyond Agreement: Diagnosing the Rationale Alignment of Automated Essay Scoring Methods based on Linguistically-informed Counterfactuals](https://mlsanity.com/paper/2405.19433/) (2024)
- [Benchmarking XAI Explanations with Human-Aligned Evaluations](https://mlsanity.com/paper/2411.02470/) (2024)
- [Improve LLM-based Automatic Essay Scoring with Linguistic Features](https://mlsanity.com/paper/2502.09497/) (2025)
- [EssayJudge: A Multi-Granular Benchmark for Assessing Automated Essay Scoring Capabilities of Multimodal Large Language Models](https://mlsanity.com/paper/2502.11916/) (2025)
- [CAFES: A Collaborative Multi-Agent Framework for Multi-Granular Multimodal Essay Scoring](https://mlsanity.com/paper/2505.13965/) (2025)
- [Teach-to-Reason with Scoring: Self-Explainable Rationale-Driven Multi-Trait Essay Scoring](https://mlsanity.com/paper/2502.20748/) (2025)
- [Opportunities and Challenges of LLMs in Education: An NLP Perspective](https://mlsanity.com/paper/2507.22753/) (2025)
- [Automated essay scoring in Arabic: a dataset and analysis of a BERT-based system](https://mlsanity.com/paper/2407.11212/) (2024)
- [FeedEval: Pedagogically Aligned Evaluation of LLM-Generated Essay Feedback](https://mlsanity.com/paper/2601.04574/) (2026)
- [Data Augmentation for Automated Essay Scoring using Transformer Models](https://mlsanity.com/paper/2210.12809/) (2022)
- [H-AES: Towards Automated Essay Scoring for Hindi](https://mlsanity.com/paper/2302.14635/) (2023)
- [Automatic Essay Multi-dimensional Scoring with Fine-tuning and Multiple Regression](https://mlsanity.com/paper/2406.01198/) (2024)
- [ChineseWebText 2.0: Large-Scale High-quality Chinese Web Text with Multi-dimensional and fine-grained information](https://mlsanity.com/paper/2411.19668/) (2024)
- [SteLLA: A Structured Grading System Using LLMs with RAG](https://mlsanity.com/paper/2501.09092/) (2025)
- [Content Significance Distribution of Sub-Text Blocks in Articles and Its Application to Article-Organization Assessment](https://mlsanity.com/paper/2311.01673/) (2023)
- [Transformer-based Joint Modelling for Automatic Essay Scoring and Off-Topic Detection](https://mlsanity.com/paper/2404.08655/) (2024)
- [LLMs Do Not Grade Essays Like Humans](https://mlsanity.com/paper/2603.23714/) (2026)
- [Tapping the Potential of Coherence and Syntactic Features in Neural Models for Automatic Essay Scoring](https://mlsanity.com/paper/2211.13373/) (2022)
- [Towards Prompt Generalization: Grammar-aware Cross-Prompt Automated Essay Scoring](https://mlsanity.com/paper/2502.08450/) (2025)
- [Towards Human-Like Grading: A Unified LLM-Enhanced Framework for Subjective Question Evaluation](https://mlsanity.com/paper/2510.07912/) (2025)
- [Enhancing Automated Essay Scoring With Three Techniques: Two-Stage Fine-Tuning, Score Alignment, and Self-Training](https://mlsanity.com/paper/2602.01747/) (2026)
- [Trait-Aware Policy Optimization for Autoregressive Multi-Trait Essay Scoring](https://mlsanity.com/paper/2605.25731/) (2026)

## Abstract (arXiv)

In recent years, pre-trained models have become dominant in most natural language processing (NLP) tasks. However, in the area of Automated Essay Scoring (AES), pre-trained models such as BERT have not been properly used to outperform other deep learning models such as LSTM. In this paper, we introduce a novel multi-scale essay representation for BERT that can be jointly learned. We also employ multiple losses and transfer learning from out-of-domain essays to further improve the performance. Experiment results show that our approach derives much benefit from joint learning of multi-scale essay representation and obtains almost the state-of-the-art result among all deep learning models in the ASAP task. Our multi-scale essay representation also generalizes well to CommonLit Readability Prize data set, which suggests that the novel text representation proposed in this paper may be a new and effective choice for long-text tasks.