# Is GPT-4 Alone Sufficient for Automated Essay Scoring?: A Comparative Judgment Approach Based on Rater Cognition

arXiv: https://arxiv.org/abs/2407.05733 · HTML: https://mlsanity.com/paper/2407.05733/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 29 references that are in the map.
- First posted on arXiv 8 Jul 2024.

## Builds on (most cited first, 3 of 3)

- [Can Large Language Models Automatically Score Proficiency of Written Essays?](https://mlsanity.com/paper/2403.06149/) (2024)
- [Autoregressive Score Generation for Multi-trait Essay Scoring](https://mlsanity.com/paper/2403.08332/) (2024)
- [DREsS: Dataset for Rubric-based Essay Scoring on EFL Writing](https://mlsanity.com/paper/2402.16733/) (2024)

## Built on by (most cited first, 7 of 7)

- [Do We Need a Detailed Rubric for Automated Essay Scoring using Large Language Models?](https://mlsanity.com/paper/2505.01035/) (2025)
- [Agreement Between Large Language Models and Human Raters in Essay Scoring: A Research Synthesis](https://mlsanity.com/paper/2512.14561/) (2025)
- [SAS-Bench: A Fine-Grained Benchmark for Evaluating Short Answer Scoring with Large Language Models](https://mlsanity.com/paper/2505.07247/) (2025)
- [Comparing Human and AI Rater Effects Using the Many-Facet Rasch Model](https://mlsanity.com/paper/2505.18486/) (2025)
- [IntelliProof: An Argumentation Network-based Conversational Helper for Organized Reflection](https://mlsanity.com/paper/2511.04528/) (2025)
- [Can Large Language Models Differentiate Harmful from Argumentative Essays? Steps Toward Ethical Essay Scoring](https://mlsanity.com/paper/2601.05545/) (2026)
- [Learning to Grade Efficiently: A Bandit-Driven Prompt-Selection Framework for Low-Cost LLM Essay Scoring](https://mlsanity.com/paper/2608.23814/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have shown promise in Automated Essay Scoring (AES), but their zero-shot and few-shot performance often falls short compared to state-of-the-art models and human raters. However, fine-tuning LLMs for each specific task is impractical due to the variety of essay prompts and rubrics used in real-world educational contexts. This study proposes a novel approach combining LLMs and Comparative Judgment (CJ) for AES, using zero-shot prompting to choose between two essays. We demonstrate that a CJ method surpasses traditional rubric-based scoring in essay scoring using LLMs.