# Unleashing Large Language Models' Proficiency in Zero-shot Essay Scoring

arXiv: https://arxiv.org/abs/2404.04941 · HTML: https://mlsanity.com/paper/2404.04941/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 52 references that are in the map.
- First posted on arXiv 7 Apr 2024.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://mlsanity.com/paper/2308.07201/) (2023)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [Benchmarking Large Language Models for News Summarization](https://mlsanity.com/paper/2301.13848/) (2023)
- [Is ChatGPT a Good NLG Evaluator? A Preliminary Study](https://mlsanity.com/paper/2303.04048/) (2023)
- [Large Language Models Are State-of-the-Art Evaluators of Translation Quality](https://mlsanity.com/paper/2302.14520/) (2023)
- [Wider and Deeper LLM Networks are Fairer LLM Evaluators](https://mlsanity.com/paper/2308.01862/) (2023)
- [Zero-shot Temporal Relation Extraction with ChatGPT](https://mlsanity.com/paper/2304.05454/) (2023)
- [Automated essay scoring with string kernels and word embeddings](https://mlsanity.com/paper/1804.07954/) (2018)
- [Many Hands Make Light Work: Using Essay Traits to Automatically Score Essays](https://mlsanity.com/paper/2102.00781/) (2021)
- [Prompt- and Trait Relation-aware Cross-prompt Essay Trait Scoring](https://mlsanity.com/paper/2305.16826/) (2023)
- [SkipFlow: Incorporating Neural Coherence Features for End-to-End Automatic Text Scoring](https://mlsanity.com/paper/1711.04981/) (2017)
- [Prompt Agnostic Essay Scorer: A Domain Generalization Approach to Cross-prompt Automated Essay Scoring](https://mlsanity.com/paper/2008.01441/) (2020)

## Built on by (most cited first, 30 of 33)

- [What are the Essential Factors in Crafting Effective Long Context Multi-Hop Instruction Datasets? Insights and Best Practices](https://mlsanity.com/paper/2409.01893/) (2024)
- [AutoSCORE: Enhancing Automated Scoring with Multi-Agent Large Language Models via Structured Component Recognition](https://mlsanity.com/paper/2509.21910/) (2025)
- [TRATES: Trait-Specific Rubric-Assisted Cross-Prompt Essay Scoring](https://mlsanity.com/paper/2505.14577/) (2025)
- [LCES: Zero-shot Automated Essay Scoring via Pairwise Comparisons Using Large Language Models](https://mlsanity.com/paper/2505.08498/) (2025)
- [EssayJudge: A Multi-Granular Benchmark for Assessing Automated Essay Scoring Capabilities of Multimodal Large Language Models](https://mlsanity.com/paper/2502.11916/) (2025)
- [CAFES: A Collaborative Multi-Agent Framework for Multi-Granular Multimodal Essay Scoring](https://mlsanity.com/paper/2505.13965/) (2025)
- [From Rubrics to Reliable Scores: Evidence-Grounded Text Evaluation with LLM Judges](https://mlsanity.com/paper/2601.08654/) (2026)
- [Teach-to-Reason with Scoring: Self-Explainable Rationale-Driven Multi-Trait Essay Scoring](https://mlsanity.com/paper/2502.20748/) (2025)
- [Rank-Then-Score: Enhancing Large Language Models for Automated Essay Scoring](https://mlsanity.com/paper/2504.05736/) (2025)
- [Automated Refinement of Essay Scoring Rubrics for Language Models via Reflect-and-Revise](https://mlsanity.com/paper/2510.09030/) (2025)
- [Quantifying the Statistical Effect of Rubric Modifications on Human-Autorater Agreement](https://mlsanity.com/paper/2605.06283/) (2026)
- [Do We Need a Detailed Rubric for Automated Essay Scoring using Large Language Models?](https://mlsanity.com/paper/2505.01035/) (2025)
- [EssayBench: Evaluating Large Language Models in Multi-Genre Chinese Essay Writing](https://mlsanity.com/paper/2506.02596/) (2025)
- [MAGIC: Multi-Agent Argumentation and Grammar Integrated Critiquer](https://mlsanity.com/paper/2506.13037/) (2025)
- [Agreement Between Large Language Models and Human Raters in Essay Scoring: A Research Synthesis](https://mlsanity.com/paper/2512.14561/) (2025)
- [Has Automated Essay Scoring Reached Sufficient Accuracy? Deriving Achievable QWK Ceilings from Classical Test Theory](https://mlsanity.com/paper/2604.19131/) (2026)
- [SAS-Bench: A Fine-Grained Benchmark for Evaluating Short Answer Scoring with Large Language Models](https://mlsanity.com/paper/2505.07247/) (2025)
- [Toward LLM-Supported Automated Assessment of Critical Thinking Subskills](https://mlsanity.com/paper/2510.12915/) (2025)
- [Enhancing Automated Essay Scoring With Three Techniques: Two-Stage Fine-Tuning, Score Alignment, and Self-Training](https://mlsanity.com/paper/2602.01747/) (2026)
- [Trait-Aware Policy Optimization for Autoregressive Multi-Trait Essay Scoring](https://mlsanity.com/paper/2605.25731/) (2026)
- [SWIM: Student Writing Simulation via Proficiency-Conditioned Generation](https://mlsanity.com/paper/2609.03215/) (2026)
- [From Struggle (06-2024) to Mastery (02-2025) LLMs Conquer Advanced Algorithm Exams and Pave the Way for Editorial Generation](https://mlsanity.com/paper/2506.04965/) (2025)
- [Leveraging AI Graders for Missing Score Imputation to Achieve Accurate Ability Estimation in Constructed-Response Tests](https://mlsanity.com/paper/2506.20119/) (2025)
- [Zero-Shot Grammar Competency Estimation Using Large Language Model Generated Pseudo Labels](https://mlsanity.com/paper/2511.13152/) (2025)
- [Automatic Essay Scoring and Feedback Generation in Basque Language Learning](https://mlsanity.com/paper/2512.08713/) (2025)
- [Automated Multiple Mini Interview (MMI) Scoring](https://mlsanity.com/paper/2602.02360/) (2026)
- [Structured Prompting for Arabic Essay Proficiency: A Trait-Centric Evaluation Approach](https://mlsanity.com/paper/2603.19668/) (2026)
- [Incentives Of EdTech: A Systematic Review Of EduNLP Research](https://mlsanity.com/paper/2606.13691/) (2026)
- [From Texts to Scores: Tracing the Emergence of Essay Quality Representations in Large Language Models](https://mlsanity.com/paper/2606.20152/) (2026)
- [When Rubrics Change: Cross-Rubric Generalization for Critical Thinking Essay Scoring](https://mlsanity.com/paper/2607.13433/) (2026)

## Abstract (arXiv)

Advances in automated essay scoring (AES) have traditionally relied on labeled essays, requiring tremendous cost and expertise for their acquisition. Recently, large language models (LLMs) have achieved great success in various tasks, but their potential is less explored in AES. In this paper, we show that our zero-shot prompting framework, Multi Trait Specialization (MTS), elicits LLMs' ample potential for essay scoring. In particular, we automatically decompose writing proficiency into distinct traits and generate scoring criteria for each trait. Then, an LLM is prompted to extract trait scores from several conversational rounds, each round scoring one of the traits based on the scoring criteria. Finally, we derive the overall score via trait averaging and min-max scaling. Experimental results on two benchmark datasets demonstrate that MTS consistently outperforms straightforward prompting (Vanilla) in average QWK across all LLMs and datasets, with maximum gains of 0.437 on TOEFL11 and 0.355 on ASAP. Additionally, with the help of MTS, the small-sized Llama2-13b-chat substantially outperforms ChatGPT, facilitating an effective deployment in real applications.