# Can Large Language Models Automatically Score Proficiency of Written Essays? arXiv: https://arxiv.org/abs/2403.06149 ยท HTML: https://mlsanity.com/paper/2403.06149/ ## Key facts - Cited by 35 papers in the map (data through 14 Oct 2026). - Builds on 5 of its 25 references that are in the map. - First posted on arXiv 10 Mar 2024. ## Builds on (most cited first, 5 of 5) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023) - [Large Language Models Are State-of-the-Art Evaluators of Translation Quality](https://mlsanity.com/paper/2302.14520/) (2023) - [On the Use of BERT for Automated Essay Scoring: Joint Learning of Multi-Scale Essay Representation](https://mlsanity.com/paper/2205.03835/) (2022) ## Built on by (most cited first, 30 of 35) - [AutoSCORE: Enhancing Automated Scoring with Multi-Agent Large Language Models via Structured Component Recognition](https://mlsanity.com/paper/2509.21910/) (2025) - [TRATES: Trait-Specific Rubric-Assisted Cross-Prompt Essay Scoring](https://mlsanity.com/paper/2505.14577/) (2025) - [Are Large Language Models Good Essay Graders?](https://mlsanity.com/paper/2409.13120/) (2024) - [Automated Text Scoring in the Age of Generative AI for the GPU-poor](https://mlsanity.com/paper/2407.01873/) (2024) - [LCES: Zero-shot Automated Essay Scoring via Pairwise Comparisons Using Large Language Models](https://mlsanity.com/paper/2505.08498/) (2025) - [Is GPT-4 Alone Sufficient for Automated Essay Scoring?: A Comparative Judgment Approach Based on Rater Cognition](https://mlsanity.com/paper/2407.05733/) (2024) - [EssayJudge: A Multi-Granular Benchmark for Assessing Automated Essay Scoring Capabilities of Multimodal Large Language Models](https://mlsanity.com/paper/2502.11916/) (2025) - [CAFES: A Collaborative Multi-Agent Framework for Multi-Granular Multimodal Essay Scoring](https://mlsanity.com/paper/2505.13965/) (2025) - [How well can LLMs Grade Essays in Arabic?](https://mlsanity.com/paper/2501.16516/) (2025) - [Teach-to-Reason with Scoring: Self-Explainable Rationale-Driven Multi-Trait Essay Scoring](https://mlsanity.com/paper/2502.20748/) (2025) - [Automatic Essay Multi-dimensional Scoring with Fine-tuning and Multiple Regression](https://mlsanity.com/paper/2406.01198/) (2024) - [SteLLA: A Structured Grading System Using LLMs with RAG](https://mlsanity.com/paper/2501.09092/) (2025) - [The NTNU System at the S&I Challenge 2025 SLA Open Track](https://mlsanity.com/paper/2506.05121/) (2025) - [Quantifying the Statistical Effect of Rubric Modifications on Human-Autorater Agreement](https://mlsanity.com/paper/2605.06283/) (2026) - [The Lazy Student's Dream: ChatGPT Passing an Engineering Course on Its Own](https://mlsanity.com/paper/2503.05760/) (2025) - [Joint Modeling of Entities and Discourse Relations for Coherence Assessment](https://mlsanity.com/paper/2509.04182/) (2025) - [Agreement Between Large Language Models and Human Raters in Essay Scoring: A Research Synthesis](https://mlsanity.com/paper/2512.14561/) (2025) - [LLMs Do Not Grade Essays Like Humans](https://mlsanity.com/paper/2603.23714/) (2026) - [Has Automated Essay Scoring Reached Sufficient Accuracy? Deriving Achievable QWK Ceilings from Classical Test Theory](https://mlsanity.com/paper/2604.19131/) (2026) - [Towards Human-Like Grading: A Unified LLM-Enhanced Framework for Subjective Question Evaluation](https://mlsanity.com/paper/2510.07912/) (2025) - [Leveraging AI Graders for Missing Score Imputation to Achieve Accurate Ability Estimation in Constructed-Response Tests](https://mlsanity.com/paper/2506.20119/) (2025) - [Can Large Language Models Differentiate Harmful from Argumentative Essays? Steps Toward Ethical Essay Scoring](https://mlsanity.com/paper/2601.05545/) (2026) - [Specialists or Generalists? Multi-Agent and Single-Agent LLMs for Essay Grading](https://mlsanity.com/paper/2601.22386/) (2026) - [Automated Multiple Mini Interview (MMI) Scoring](https://mlsanity.com/paper/2602.02360/) (2026) - [From Prompting to Preference Optimization: A Comparative Study of LLM-based Automated Essay Scoring](https://mlsanity.com/paper/2603.06424/) (2026) - [Structured Prompting for Arabic Essay Proficiency: A Trait-Centric Evaluation Approach](https://mlsanity.com/paper/2603.19668/) (2026) - [Using Large Language Models to Support High Volume Application Review for an Undergraduate Research Program](https://mlsanity.com/paper/2606.05564/) (2026) - [MADRAG: Multi-Agent Debate with Retrieval-Augmented Generation for Training-Free Analytic Essay Scoring](https://mlsanity.com/paper/2606.06754/) (2026) - [The Order Matters: Sequential Fine-Tuning of LLaMA for Coherent Automated Essay Scoring](https://mlsanity.com/paper/2606.10327/) (2026) - [LLM Performance on a Real, Double-Marked GCSE Benchmark](https://mlsanity.com/paper/2606.24973/) (2026) ## Abstract (arXiv) Although several methods were proposed to address the problem of automated essay scoring (AES) in the last 50 years, there is still much to desire in terms of effectiveness. Large Language Models (LLMs) are transformer-based models that demonstrate extraordinary capabilities on various tasks. In this paper, we test the ability of LLMs, given their powerful linguistic knowledge, to analyze and effectively score written essays. We experimented with two popular LLMs, namely ChatGPT and Llama. We aim to check if these models can do this task and, if so, how their performance is positioned among the state-of-the-art (SOTA) models across two levels, holistically and per individual writing trait. We utilized prompt-engineering tactics in designing four different prompts to bring their maximum potential to this task. Our experiments conducted on the ASAP dataset revealed several interesting observations. First, choosing the right prompt depends highly on the model and nature of the task. Second, the two LLMs exhibited comparable average performance in AES, with a slight advantage for ChatGPT. Finally, despite the performance gap between the two LLMs and SOTA models in terms of predictions, they provide feedback to enhance the quality of the essays, which can potentially help both teachers and students.