# Human-AI Collaborative Essay Scoring: A Dual-Process Framework with LLMs

arXiv: https://arxiv.org/abs/2401.06431 · HTML: https://mlsanity.com/paper/2401.06431/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 47 references that are in the map.
- First posted on arXiv 12 Jan 2024.

## Builds on (most cited first, 15 of 15)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [REPLUG: Retrieval-Augmented Black-Box Language Models](https://mlsanity.com/paper/2301.12652/) (2023)
- [In-Context Retrieval-Augmented Language Models](https://mlsanity.com/paper/2302.00083/) (2023)
- [Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance](https://mlsanity.com/paper/2006.14779/) (2020)
- [Can language models learn from explanations in context?](https://mlsanity.com/paper/2204.02329/) (2022)
- [Making Large Language Models Better Reasoners with Step-Aware Verifier](https://mlsanity.com/paper/2206.02336/) (2022)
- [SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks](https://mlsanity.com/paper/2305.17390/) (2023)
- [On the Use of BERT for Automated Essay Scoring: Joint Learning of Multi-Scale Essay Representation](https://mlsanity.com/paper/2205.03835/) (2022)
- [Language models and Automated Essay Scoring](https://mlsanity.com/paper/1909.09482/) (2019)
- [Prompt Agnostic Essay Scorer: A Domain Generalization Approach to Cross-prompt Automated Essay Scoring](https://mlsanity.com/paper/2008.01441/) (2020)
- [Designing for human-AI complementarity in K-12 education](https://mlsanity.com/paper/2104.01266/) (2021)
- [Crowd Score: A Method for the Evaluation of Jokes using Large Language Model AI Voters as Judges](https://mlsanity.com/paper/2212.11214/) (2022)
- [DUMA: a Dual-Mind Conversational Agent with Fast and Slow Thinking](https://mlsanity.com/paper/2310.18075/) (2023)

## Built on by (most cited first, 30 of 38)

- [Multi-Agent Debate for LLM Judges with Adaptive Stability Detection](https://mlsanity.com/paper/2510.12697/) (2025)
- [Automated Text Scoring in the Age of Generative AI for the GPU-poor](https://mlsanity.com/paper/2407.01873/) (2024)
- [Beyond Agreement: Diagnosing the Rationale Alignment of Automated Essay Scoring Methods based on Linguistically-informed Counterfactuals](https://mlsanity.com/paper/2405.19433/) (2024)
- [EssayJudge: A Multi-Granular Benchmark for Assessing Automated Essay Scoring Capabilities of Multimodal Large Language Models](https://mlsanity.com/paper/2502.11916/) (2025)
- [CAFES: A Collaborative Multi-Agent Framework for Multi-Granular Multimodal Essay Scoring](https://mlsanity.com/paper/2505.13965/) (2025)
- [Position: LLMs Can be Good Tutors in English Education](https://mlsanity.com/paper/2502.05467/) (2025)
- [KatFishNet: Detecting LLM-Generated Korean Text through Linguistic Feature Analysis](https://mlsanity.com/paper/2503.00032/) (2025)
- [Rank-Then-Score: Enhancing Large Language Models for Automated Essay Scoring](https://mlsanity.com/paper/2504.05736/) (2025)
- [Does the Prompt-based Large Language Model Recognize Students' Demographics and Introduce Bias in Essay Scoring?](https://mlsanity.com/paper/2504.21330/) (2025)
- [DITTO: A Spoofing Attack Framework on Watermarked LLMs via Knowledge Distillation](https://mlsanity.com/paper/2510.10987/) (2025)
- [Benchmarking Educational LLMs with Analytics: A Case Study on Gender Bias in Feedback](https://mlsanity.com/paper/2511.08225/) (2025)
- [Exploring the Potential of Large Multimodal Models as Effective Alternatives for Pronunciation Assessment](https://mlsanity.com/paper/2503.11229/) (2025)
- [Exploring LLM Autoscoring Reliability in Large-Scale Writing Assessments Using Generalizability Theory](https://mlsanity.com/paper/2507.19980/) (2025)
- [Quantifying the Statistical Effect of Rubric Modifications on Human-Autorater Agreement](https://mlsanity.com/paper/2605.06283/) (2026)
- [On the Suitability of pre-trained foundational LLMs for Analysis in German Legal Education](https://mlsanity.com/paper/2412.15902/) (2024)
- [Agreement Between Large Language Models and Human Raters in Essay Scoring: A Research Synthesis](https://mlsanity.com/paper/2512.14561/) (2025)
- [Machine-Assisted Grading of Nationwide School-Leaving Essay Exams with LLMs and Statistical NLP](https://mlsanity.com/paper/2601.16314/) (2026)
- [From Motion Signals to Insights: A Unified Framework for Student Behavior Analysis and Feedback in Physical Education Classes](https://mlsanity.com/paper/2503.06525/) (2025)
- [Cyborg Data: Merging Human with AI Generated Training Data](https://mlsanity.com/paper/2503.22736/) (2025)
- [Beyond the Score: Uncertainty-Calibrated LLMs for Automated Essay Assessment](https://mlsanity.com/paper/2509.15926/) (2025)
- [Comparison of Scoring Rationales Between Large Language Models and Human Raters](https://mlsanity.com/paper/2509.23412/) (2025)
- [Toward LLM-Supported Automated Assessment of Critical Thinking Subskills](https://mlsanity.com/paper/2510.12915/) (2025)
- [PEMUTA: Pedagogically-Enriched Multi-Granular Undergraduate Thesis Assessment](https://mlsanity.com/paper/2507.19556/) (2025)
- [ZPD-SCA: Unveiling the Blind Spots of LLMs in Assessing Students' Cognitive Abilities](https://mlsanity.com/paper/2508.14377/) (2025)
- [TransGAT: Transformer-Based Graph Neural Networks for Multi-Dimensional Automated Essay Scoring](https://mlsanity.com/paper/2509.01640/) (2025)
- [Scaling Equitable Reflection Assessment in Education via Large Language Models and Role-Based Feedback Agents](https://mlsanity.com/paper/2511.11772/) (2025)
- [Applying Large Language Models to Characterize Public Narratives](https://mlsanity.com/paper/2511.13505/) (2025)
- [Automatic Essay Scoring and Feedback Generation in Basque Language Learning](https://mlsanity.com/paper/2512.08713/) (2025)
- [S-GRADES -- Studying Generalization of Student Response Assessments in Diverse Evaluative Settings](https://mlsanity.com/paper/2603.10233/) (2026)
- [Understanding Teacher Revisions of Large Language Model-Generated Feedback](https://mlsanity.com/paper/2603.27806/) (2026)

## Abstract (arXiv)

Receiving timely and personalized feedback is essential for second-language learners, especially when human instructors are unavailable. This study explores the effectiveness of Large Language Models (LLMs), including both proprietary and open-source models, for Automated Essay Scoring (AES). Through extensive experiments with public and private datasets, we find that while LLMs do not surpass conventional state-of-the-art (SOTA) grading models in performance, they exhibit notable consistency, generalizability, and explainability. We propose an open-source LLM-based AES system, inspired by the dual-process theory. Our system offers accurate grading and high-quality feedback, at least comparable to that of fine-tuned proprietary LLMs, in addition to its ability to alleviate misgrading. Furthermore, we conduct human-AI co-grading experiments with both novice and expert graders. We find that our system not only automates the grading process but also enhances the performance and efficiency of human graders, particularly for essays where the model has lower confidence. These results highlight the potential of LLMs to facilitate effective human-AI collaboration in the educational context, potentially transforming learning experiences through AI-generated feedback.