SAGE: Hierarchical LLM-Based Literary Evaluation through Ontology-Grounded Interpretive Dimensions
Not on the map. Why?
Abstract (arXiv)
Evaluating literary quality requires assessing interpretive dimensions such as cultural representation, emotional depth, and philosophical sophistication that resist straightforward computational measurement. We introduce SAGE, a hierarchical evaluation framework that decomposes literary quality into ontology-grounded interpretive dimensions assessed through structured large language model evaluation with multi-round iterative reflection and independent validation. We validate the framework on 100 short stories (50 canonical works, 30 pulp fiction, 20 LLM-generated narratives) across three analytical layers (cultural, emotional-psychological, existential-philosophical) using dual-mode assessment. Across 600 evaluations, the framework achieves 98.8% score convergence and greater than 94% inter-rater agreement, with near-perfect mode invariance between content-based and metadata-based evaluation. Statistical analysis reveals a consistent genre hierarchy (Canonical > Pulp > LLM, all p<0.001) with layer-specific discrimination: cultural critique and philosophical depth exhibit very large effect sizes (Cohen's d>2.4), while emotional representation shows smaller gaps (d=1.68), suggesting that affective patterns are more learnable from training data than critical stance or philosophical depth. Cross-layer correlations (r=0.649-0.683) confirm the three dimensions capture empirically distinguishable quality facets. These findings demonstrate that theory-driven LLM evaluation can achieve measurement-grade reliability and support systematic identification of where current generative models fall short of human literary production, with direct implications for scalable automated evaluation of open-ended text generation.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 16 of its 58 references that are in the map. First posted on arXiv 8 May 2026.
Builds on
16 of 58 references.
References · 16
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
- 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
- 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
- 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 377
- 2017Why We Need New Evaluation Metrics for NLGcited by 232
- 2022Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionalscited by 191
- 2019Emotion Recognition in Conversation: Research Challenges, Datasets, and Recent Advancescited by 118
- 2022Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generationcited by 73
- 2016The emotional arcs of stories are dominated by six basic shapescited by 69
- 2024Agents' Room: Narrative Generation through Multi-step Collaborationcited by 57
- 2025SCORE: Story Coherence and Retrieval Enhancement for AI Narrativescited by 42
- 2024What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluationcited by 17
4 more
- 2025A Cognitive Writing Perspective for Constrained Long-Form Text Generationcited by 14
- 2025LongEval: A Comprehensive Analysis of Long-Text Generation Through a Plan-based Paradigmcited by 14
- 2025NarraBench: A Comprehensive Framework for Narrative Benchmarkingcited by 13
- 2025BookWorld: From Novels to Interactive Agent Societies for Creative Story Generationcited by 12
Built on by
Not cited yet.