mlsanityData to 14 Oct 2026

Help Me Write a Story: Evaluating LLMs' Ability to Generate Writing Feedback

arXiv 2507.16007 ↗First posted 21 Jul 2025cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Can LLMs provide support to creative writers by giving meaningful writing feedback? In this paper, we explore the challenges and limitations of model-generated writing feedback by defining a new task, dataset, and evaluation frameworks. To study model performance in a controlled manner, we present a novel test set of 1,300 stories that we corrupted to intentionally introduce writing issues. We study the performance of commonly used LLMs in this task with both automatic and human evaluation metrics. Our analysis shows that current models have strong out-of-the-box behavior in many respects -- providing specific and mostly accurate writing feedback. However, models often fail to identify the biggest writing issue in the story and to correctly decide when to offer critical vs. positive feedback.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 28 of its 45 references that are in the map. First posted on arXiv 21 Jul 2025.

Builds on

28 of 45 references.

References · 28

  1. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  2. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  3. 2018Hierarchical Neural Story Generationcited by 1,193
  4. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 417
  5. 2022CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilitiescited by 249
  6. 2019Leveraging Pre-trained Checkpoints for Sequence Generation Taskscited by 182
  7. 2023Art or Artifice? Large Language Models and the False Promise of Creativitycited by 172
  8. 2018A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applicationscited by 136
  9. 2024MARG: Multi-Agent Review Generation for Scientific Paperscited by 104
  10. 2021Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Textcited by 89
  11. 2022Help me write a poem: Instruction Tuning as a Vehicle for Collaborative Poetry Writingcited by 71
  12. 2022Creative Writing with an AI-Powered Writing Assistant: Perspectives from Professional Writerscited by 69
16 more
  1. 2023RewriteLM: An Instruction-Tuned Large Language Model for Text Rewritingcited by 67
  2. 2022NLPeer: A Unified Resource for the Computational Study of Peer Reviewcited by 60
  3. 2022Understanding Iterative Revision from Human-Written Textcited by 57
  4. 2024Agents' Room: Narrative Generation through Multi-step Collaborationcited by 57
  5. 2024Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approachcited by 54
  6. 2022Read, Revise, Repeat: A System Demonstration for Human-in-the-loop Iterative Text Revisioncited by 46
  7. 2023Writer-Defined AI Personas for On-Demand Feedback Generationcited by 36
  8. 2023Beyond the Chat: Executable and Verifiable Text-Editing with LLMscited by 35
  9. 2024Automated Focused Feedback Generation for Scientific Writing Assistancecited by 33
  10. 2023LLM-as-a-tutor in EFL Writing Education: Focusing on Evaluation of Student-LLM Interactioncited by 29
  11. 2018WikiAtomicEdits: A Multilingual Corpus of Wikipedia Edits for Modeling Language and Discoursecited by 26
  12. 2025Finding Flawed Fictions: Evaluating Complex Reasoning in Language Models via Plot Hole Detectioncited by 24
  13. 2022Improving Iterative Text Revision by Learning Where to Edit from Other Revision Taskscited by 20
  14. 2021ePiC: Employing Proverbs in Context as a Benchmark for Abstract Language Understandingcited by 14
  15. 2024From Pen to Prompt: How Creative Writers Integrate AI into their Writing Practicecited by 14
  16. 2024LLMCRIT: Teaching Large Language Models to Use Criteriacited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 10.

11 papers.

  1. 2026LLMs Exhibit Significantly Lower Uncertainty in Creative Writing Than Professional Writerscited by 4
  2. 2025Accepted with Minor Revisions: Value of AI-Assisted Scientific Writingcited by 1
  3. 2026REFINE: Real-world Exploration of Interactive Feedback and Student Behaviourcited by 1
  4. 2026Debate Helps Weak Judges Reward Stronger Modelscited by 1
  5. 2026The Crutch or the Ceiling? How Different Generations of LLMs Shape EFL Student Writingscited by 0
  6. 2026FOXGLOVE: Understanding Goal-Oriented and Anchored Writing Feedback from Experts and LLMs on Argumentative Essayscited by 0
  7. 2026ArguLens: An Open-Source System for Automated Essay Scoring and Label-Aware Feedback Generationcited by 0
  8. 2026Generating Constructive Feedback on Stories via Reinforcement Learningcited by 0
  9. 2026Evaluating Feedback Focus and Pedagogical Adaptivity in LLM-Generated Feedback on Student Writingcited by 0
  10. 2026Adversarial Closed-Loop Curriculum for Evolving Role-Playing Agentscited by 0
  11. 2026Scaling Long-Form Story Generation via Narrative State Trackingcited by 0