mlsanityData to 14 Oct 2026

When Reasoning Supervision Hurts: TTCW-Based Long-Form Literary Review Generation

arXiv 2605.20364 ↗First posted 19 May 2026cs.CLcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Automatic evaluation of long-form literary writing remains challenging, particularly for creativity-related dimensions such as originality and flexibility. We address the lack of large-scale supervision for structured Torrance Test of Creative Writing (TTCW)-based evaluation by constructing a dataset of 263,911 long-form stories, each paired with scalar scores and meta-synthesised review comments across 14 TTCW-based dimensions. Using this dataset, we study whether reviewer-style reasoning traces improve structured literary review generation across Qwen3-4B, Qwen3-8B, and Nemotron-8B. Across all three models, no-trace supervision yields higher score imitation fidelity and near-complete report generation, whereas trace supervision reduces completion to 85.9--90.9%. Failure analysis shows that 88.6% of failed reasoning generations contain semantic repetition loops inside the trace. These loops concentrate in particular story--criterion interactions and progressively harden after onset, while comparable repetition is rare in the supervision data. Our results suggest that, for long-form tasks requiring strictly structured outputs, explicit reasoning traces can introduce generation instability that outweighs their overall benefit.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 8 of its 26 references that are in the map. First posted on arXiv 19 May 2026.

Builds on

8 of 26 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models8 Aug
  2. 2025Qwen3 Technical Report14 May

Other references · 6

  1. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  2. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 872
  3. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 316
  4. 2025Llama-Nemotron: Efficient Reasoning Modelscited by 158
  5. 2025WritingBench: A Comprehensive Benchmark for Generative Writingcited by 106
  6. 2025NVIDIA Nemotron 3: Efficient and Open Intelligencecited by 65

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writing
  2. 2025CreativityPrism: A Cross-Domain Evaluation Framework for Large Language Model Creativity
  3. 2025Curiosity-Driven LLM-as-a-judge for Personalized Creative Judgment
  4. 2026Towards Human-Level Book-Writing Capability
  5. 2025Learning to Reason for Long-Form Story Generation
  6. 2025CoKe: Customizable Fine-Grained Story Evaluation via Chain-of-Keyword Rationalization
  7. 2026QUIET: A Multi-Blank Cascaded Story Cloze Benchmark for LLM Creative Generation Capability
  8. 2026CLIN: an Objective Framework for Evaluating Creativity in Short Persian Literary Text