mlsanityData to 7 Oct 2026

LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writing

arXiv 2507.00769 ↗First posted 1 Jul 2025cs.CL cs.AIcited by 39

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Evaluating creative writing generated by large language models (LLMs) remains challenging because open-ended narratives lack ground truths. Without performant automated evaluation methods, off-the-shelf (OTS) language models are employed as zero-shot judges, yet their reliability is unclear in this context. In pursuit of robust evaluation for creative writing, we introduce LitBench, the first standardized benchmark and paired dataset for creative writing verification, comprising a held-out test set of 2,480 debiased, human-labeled story comparisons drawn from Reddit and a 43,827-pair training corpus of human preference labels. Using LitBench, we (i) benchmark zero-shot LLM judges, (ii) train Bradley Terry and generative reward models, and (iii) conduct an online human study to validate reward model rankings on newly LLM-generated stories. Our benchmark identifies Claude-3.7-Sonnet as the strongest off-the-shelf judge, reaching 73% agreement with human preferences; among trained reward models, Bradley-Terry and Generative reward models both attain an accuracy of 78%, outperforming all off-the-shelf judges. An online human study further confirms that our trained reward models consistently align with human preferences in novel LLM-generated stories. We release LitBench and reward models at https://huggingface.co/collections/SAA-Lab/litbench-68267b5da3aafe58f9e43461, providing a vetted resource for reliable, automated evaluation and optimization of creative writing systems.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 45 references that are in the map. First posted on arXiv 1 Jul 2025.

Builds on

24 of 45 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Modifying Large Language Model Post-Training for Diverse Creative Writing21 Mar
  2. 2024Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking23 Sep
  3. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates23 Aug
  4. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  5. 2020Learning to summarize from human feedback2 Sep

Other references · 19

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  7. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  8. 2018Hierarchical Neural Story Generationcited by 1,182
  9. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  10. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  11. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  12. 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 300
7 more
  1. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  2. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 277
  3. 2023Art or Artifice? Large Language Models and the False Promise of Creativitycited by 168
  4. 2024Generative Reward Modelscited by 148
  5. 2019Jointly Measuring Diversity and Quality in Text Generation Modelscited by 61
  6. 2025Automated Creativity Evaluation for Large Language Models: A Reference-Based Approachcited by 20
  7. 2020On the Relation between Quality-Diversity Evaluation and Distribution-Fitting Goal in Text Generationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 31.

39 papers.

Sort:

By area:

  1. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 49
  2. 2025The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judgecited by 13
  3. 2026Reward Modeling from Natural Language Human Feedbackcited by 7
  4. 2025Beyond Correctness: Evaluating Subjective Writing Preferences Across Culturescited by 6
  5. 2026CREATE: Testing LLMs for Associative Creativitycited by 6
  6. 2025Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMscited by 5
  7. 2026The Judge Who Never Admits: Hidden Shortcuts in LLM-based Evaluationcited by 3
  8. 2026Beyond the Illusion of Consensus: From Surface Heuristics to Knowledge-Grounded Evaluation in LLM-as-a-Judgecited by 3
  9. 2025DramaBench: A Six-Dimensional Evaluation Framework for Drama Script Continuationcited by 2
  10. 2026Rewarding Creativity: A Human-Aligned Generative Reward Model for Reinforcement Learning in Storytellingcited by 2
  11. 2026Polaris: A G"odel Agent Framework for Small Language Models through Experience-Abstracted Policy Repaircited by 2
  12. 2026Are LLMs becoming similarly creative? Evidence from three years of modelscited by 2
27 more
  1. 2026Interrupting the Loop: Periodic Subject Changes Raise Judged Surprise and Connection in Base Language Modelscited by 2
  2. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 2
  3. 2025CineLOG: A Training Free Approach for Cinematic Long Video Generationcited by 1
  4. 2025Casting a SPELL: Sentence Pairing Exploration for LLM Limitation-breakingcited by 1
  5. 2026MedConclusion: A Benchmark for Biomedical Conclusion Generation from Structured Abstractscited by 1
  6. 2026StoryAlign: Evaluating and Training Reward Models for Story Generationcited by 1
  7. 2026Argument Collapse: LLMs Flatten Long-Form Public Debatecited by 1
  8. 2026Democratic ICAI: Debating Our Way to Steering Principles from Preferencescited by 1
  9. 2026When Stories Evolve: Benchmarking LLM Storytelling Across Agent Architectures in Open-Ended World Simulationscited by 1
  10. 2026Self- and Other-Labels Induce Bidirectional Bias in LLM Judgescited by 1
  11. 2025COIG-Writer: A High-Quality Dataset for Chinese Creative Writing with Thought Processescited by 0
  12. 2025Capturing Classic Authorial Style in Long-Form Story Generation with GRPO Fine-Tuningcited by 0
  13. 2026Beyond Model Scaling: Test-Time Intervention for Efficient Deep Reasoningcited by 0
  14. 2026Dynamic Delayed Tree Expansion For Improved Multi-Path Speculative Decodingcited by 0
  15. 2026PlotTwist: A Creative Plot Generation Framework with Small Language Modelscited by 0
  16. 2026UniCreative: Unifying Long-form Logic and Short-form Sparkle via Reference-Free Reinforcement Learningcited by 0
  17. 2026Reasoning4Sciences: Bridging Reasoning Language Models to All Scientific Branchescited by 0
  18. 2026POLARIS: Guiding Small Models to Write Long Storiescited by 0
  19. 2026GENIE: A Fine-Grained Measure for Noveltycited by 0
  20. 2026Can Language Models Imagine Without Seeing? Ekphrasis: Measuring Visual Creative Ideation in Text-Only LLMscited by 0
  21. 2026Scaling Creative Writing Beyond Story-Centric Data with Attribute-Guided Genre Expansioncited by 0
  22. 2026CLIN: an Objective Framework for Evaluating Creativity in Short Persian Literary Textcited by 0
  23. 2026EvoRS: On-Policy Self-Evolution of Reward Systems for Open-Ended Reinforcement Learningcited by 0
  24. 2026Agents Can Use Base Models to Evade AI Detectioncited by 0
  25. 2026The Effects of Incremental Instruction Delivery on Language-Model Creative Writingcited by 0
  26. 2026SlopBench: How Well Can We Rank Language Models by Slop? A Multi-Domain Benchmark of Repetitive AI Writingcited by 0
  27. 2026Scaling Long-Form Story Generation via Narrative State Trackingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Curiosity-Driven LLM-as-a-judge for Personalized Creative Judgment
  2. 2026When Reasoning Supervision Hurts: TTCW-Based Long-Form Literary Review Generation
  3. 2025AI-Slop to AI-Polish? Aligning Language Models through Edit-Based Writing Rewards and Test-time Computation
  4. 2026Towards Human-Level Book-Writing Capability
  5. 2026QUIET: A Multi-Blank Cascaded Story Cloze Benchmark for LLM Creative Generation Capability
  6. 2025CreativityPrism: A Cross-Domain Evaluation Framework for Large Language Model Creativity
  7. 2025Creative Preference Optimization
  8. 2025Creativity Benchmark: A benchmark for marketing creativity for large language models