# When LLM Essays Outscore Student Essays: What a Korean Writing Rubric Rewards and Where Readers Disagree

arXiv: https://arxiv.org/abs/2601.19913 · HTML: https://mlsanity.com/paper/2601.19913/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 22 references that are in the map.
- First posted on arXiv 6 Jan 2026.

## Builds on (most cited first, 7 of 7)

- [All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text](https://mlsanity.com/paper/2107.00061/) (2021)
- [Human heuristics for AI-generated language are flawed](https://mlsanity.com/paper/2206.07271/) (2022)
- [Real or Fake Text?: Investigating Human Ability to Detect Boundaries Between Human-Written and Machine-Generated Text](https://mlsanity.com/paper/2212.12672/) (2022)
- [People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text](https://mlsanity.com/paper/2501.15654/) (2025)
- [The Reader is the Metric: How Textual Features and Reader Profiles Explain Conflicting Evaluations of AI Creative Writing](https://mlsanity.com/paper/2506.03310/) (2025)
- [KatFishNet: Detecting LLM-Generated Korean Text through Linguistic Feature Analysis](https://mlsanity.com/paper/2503.00032/) (2025)
- [Linguistic and Embedding-Based Profiling of Texts generated by Humans and Large Language Models](https://mlsanity.com/paper/2507.13614/) (2025)

## Abstract (arXiv)

LLMs now help students plan, draft, and revise essays. Educational assessment therefore faces a basic question: how should student and LLM writing be compared? Rubrics assign points to content, organization, and expression. Their total can still hide which criteria drive the comparison, where ratings approach the maximum, and where readers disagree. We therefore conducted a secondary, post hoc audit of a Korean writing study with source-informed scoring. Three Korean language and literature majors used a 16-criterion, 100-point rubric to score six student essays and 24 essays from four LLMs prompted for three school levels. After source disclosure, they helped develop the rubric and, according to the protocol, scored the Phase 2 essays without gold labels while recording source judgments. The pooled LLM mean exceeds the student mean by 18.35 points. Orthographic norms and genre-appropriate register contribute 8.19 points, or 44.7% of the gap. All 72 LLM ratings reach the orthographic maximum, and 70 reach the register maximum. The student source mean is highest or tied highest on both creativity criteria and natural Korean phrasing, where reader agreement is weak. Criterion-level auditing therefore offers a deeper account of human and LLM writing than the total alone.