What is Good? Extracting and Testing Implicit Theories of Literary Quality from LLM Reasoning Traces
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
What makes writing "good" remains a persistent question in literary studies and computational linguistics. We present a two-study investigation of how reasoning-enabled LLMs evaluate literary quality. In Study 1, we construct a benchmark of 30 real texts spanning six quality tiers, from canonical literature to anonymous forum posts, and extract the model's implicit theory of quality from its reasoning traces. Across five DeepSeek replications, the model achieves 79.3% mean tier-classification accuracy. The traces reveal a consistent stated theory: the model values intentionality over correctness, prioritizing craft, depth, and distinctive voice. A familiarity experiment with style-matched but unrecognizable passages suggests that source recognition may inflate scores, although this is confounded by genuine quality differences between canonical originals and researcher-written pastiches. In Study 2, we probe this theory through systematic degradation of five canonical prose passages. We apply six manipulations - vocabulary simplification, rhythm flattening, imagery removal, voice genericization, structure simplification, and combined degradation - and reevaluate each version. Vocabulary simplification causes the smallest quality loss (0.41 +/- 0.46 points), far below structure (2.78) or voice (2.34) loss. Combined degradation is devastating (-5.64) but subadditive. An exploratory comparison with Qwen QwQ shows the same broad qualitative pattern. Together, these studies suggest that LLM judgments of writing quality are holistic, author-specific, and more sensitive to structural than lexical features, with implications for automated writing feedback and computational aesthetics.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 4 of its 18 references that are in the map. First posted on arXiv 7 Apr 2026.
Builds on
4 of 18 references.
References · 4
- 2023Large Language Models are not Fair Evaluatorscited by 1,085
- 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 703
- 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
- 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 150
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025AI-Slop to AI-Polish? Aligning Language Models through Edit-Based Writing Rewards and Test-time Computation
- 2025KRISTEVA: Close Reading as a Novel Task for Benchmarking Interpretive Reasoning
- 2025Literary Evidence Retrieval via Long-Context Language Models
- 2026From annotation to reasoning: Culture in language models
- 2026When Reasoning Supervision Hurts: TTCW-Based Long-Form Literary Review Generation
- 2025LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writing
- 2026Calibrated Surprise: An Information-Theoretic Account of Creative Quality
- 2026Towards Human-Level Book-Writing Capability