mlsanityData to 7 Oct 2026

WritingBench: A Comprehensive Benchmark for Generative Writing

arXiv 2503.05244 ↗First posted 7 Mar 2025cs.AI cs.CLcited by 95

Not on the map. Why?

Cited by 21 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent advancements in large language models (LLMs) have significantly enhanced text generation capabilities, yet evaluating their performance in generative writing remains a challenge. Existing benchmarks primarily focus on generic text generation or limited in writing tasks, failing to capture the diverse requirements of high-quality written contents across various domains. To bridge this gap, we present WritingBench, a comprehensive benchmark designed to evaluate LLMs across 6 core writing domains and 100 subdomains. We further propose a query-dependent evaluation framework that empowers LLMs to dynamically generate instance-specific assessment criteria. This framework is complemented by a fine-tuned critic model for criteria-aware scoring, enabling evaluations in style, format and length. The framework's validity is further demonstrated by its data curation capability, which enables a 7B-parameter model to outperform the performance of GPT-4o in writing. We open-source the benchmark, along with evaluation tools and modular framework components, to advance the development of LLMs in writing.

Key facts. Cited by 95 papers in the map (data through 7 Oct 2026). Builds on 20 of its 31 references that are in the map. First posted on arXiv 7 Mar 2025.

Builds on

20 of 31 references.

References · 20

  1. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  2. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  3. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  4. 2022Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionalscited by 191
  5. 2024Large Language Models for Education: A Survey and Outlookcited by 185
  6. 2024Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Modelscited by 150
  7. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 142
  8. 2023A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writingcited by 98
  9. 2024One Thousand and One Pairs: A "novel" challenge for long-context language modelscited by 80
  10. 2023EQ-Bench: An Emotional Intelligence Benchmark for Large Language Modelscited by 75
  11. 2023SuperCLUE: A Comprehensive Chinese Large Language Model Benchmarkcited by 50
  12. 2024HelloBench: Evaluating Long Text Generation Capabilities of Large Language Modelscited by 42
8 more
  1. 2024Suri: Multi-constraint Instruction Following for Long-form Text Generationcited by 34
  2. 2024Pron vs Prompt: Can Large Language Models already Challenge a World-Class Fiction Author at Creative Text Writing?cited by 26
  3. 2024Weaver: Foundation Models for Creative Writingcited by 25
  4. 2023Adapting Large Language Models for Education: Foundational Capabilities, Potentials, and Challengescited by 13
  5. 2024MoPS: Modular Story Premise Synthesis for Open-Ended Automatic Story Generationcited by 13
  6. 2024A Sentiment Consolidation Framework for Meta-Review Generationcited by 10
  7. 2024Ex3: Automatic Novel Writing by Extracting, Excelsior and Expandingcited by 10
  8. 2024Fennec: Fine-grained Language Model Evaluation and Correction Extended through Branching and Bridgingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 38; 2026: 57.

95 papers.

Sort:

By area:

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2025Qwen3-VL Technical Reportcited by 2,534
  3. 2025Qwen3-Omni Technical Reportcited by 451
  4. 2025DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agentscited by 236
  5. 2025Reinforcement Learning with Rubric Anchorscited by 108
  6. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 56
  7. 2025MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systemscited by 56
  8. 2025Baichuan-M2: Scaling Medical Capability with Large Verifier Systemcited by 55
  9. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 49
  10. 2025Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewardscited by 47
  11. 2025Reverse-Engineered Reasoning for Open-Ended Generationcited by 32
  12. 2025DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecyclecited by 22
38 more
  1. 2026Reward Hacking in Rubric-Based Reinforcement Learningcited by 18
  2. 2025Evaluating LLM Metrics Through Real-World Capabilitiescited by 17
  3. 2026ArenaRL: Scaling RL for Open-Ended Agents via Tournament-based Relative Rankingcited by 17
  4. 2025LongWriter-Zero: Mastering Ultra-Long Text Generation via Reinforcement Learningcited by 13
  5. 2025Reinforcement Learning Foundations for Deep Research Systems: A Surveycited by 13
  6. 2025EvalAgent: Discovering Implicit Evaluation Criteria from the Webcited by 11
  7. 2025RLMR: Reinforcement Learning with Mixed Rewards for Creative Writingcited by 10
  8. 2025Evaluating Text Creativity across Diverse Domains: A Dataset and Large Language Model Evaluatorcited by 9
  9. 2025ACE-RL: Adaptive Constraint-Enhanced Reward for Long-form Generation Reinforcement Learningcited by 9
  10. 2025A Survey of AI Scientistscited by 9
  11. 2026Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learningcited by 9
  12. 2026Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifierscited by 9
  13. 2026SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decodingcited by 7
  14. 2026EvoRubric: Self-Evolving Rubric-Driven RL for Open-Ended Generationcited by 7
  15. 2025Understanding DeepResearch via Reportscited by 6
  16. 2025Instructions are all you need: Self-supervised Reinforcement Learning for Instruction Followingcited by 6
  17. 2025Beyond Correctness: Evaluating Subjective Writing Preferences Across Culturescited by 6
  18. 2025SuperWriter: Reflection-Driven Long-Form Generation with Large Language Modelscited by 5
  19. 2025D2HScore: Reasoning-Aware Hallucination Detection via Semantic Breadth and Depth Analysis in LLMscited by 5
  20. 2025The Personalization Trap: How User Memory Alters Emotional Reasoning in LLMscited by 5
  21. 2026AgentSPEX: An Agent SPecification and EXecution Languagecited by 5
  22. 2025Zero Reinforcement Learning Towards General Domainscited by 4
  23. 2025Extending RLVR to Open-Ended Tasks via Verifiable Multiple-Choice Reformulationcited by 4
  24. 2025CoSineVerifier: Tool-Augmented Answer Verification for Computation-Oriented Scientific Questionscited by 4
  25. 2026Mach-Mind-4-Flash Technical Reportcited by 4
  26. 2026CriPO: Enhancing Rubric-based RL via Self-Distillationcited by 4
  27. 2025Beyond the Trade-off: Self-Supervised Reinforcement Learning for Reasoning Models' Instruction Followingcited by 3
  28. 2025SGSimEval: A Comprehensive Multifaceted and Similarity-Enhanced Benchmark for Automatic Survey Generation Systemscited by 3
  29. 2026IDRBench: Benchmarking the Interactive Capabilities of Deep Research Agentscited by 3
  30. 2026From Absolute to Relative: Rethinking Reward Shaping in Group-Based Reinforcement Learningcited by 3
  31. 2026Beyond the Illusion of Consensus: From Surface Heuristics to Knowledge-Grounded Evaluation in LLM-as-a-Judgecited by 3
  32. 2026R2-Write: Reflection and Revision for Open-Ended Writing with Deep Reasoningcited by 3
  33. 2026The Format Taxcited by 3
  34. 2026ARES: Automated Rubric Synthesis for Scalable LLM Reinforcement Learningcited by 3
  35. 2026Focal Reward: Balanced Reinforcement Learning under Rubric-Based Rewardscited by 3
  36. 2026A Local Perturbation Theory for Cross-Domain Interference and Recovery in Multi-Domain RLcited by 3
  37. 2026Full-bandwidth transformercited by 3
  38. 2025EssayBench: Evaluating Large Language Models in Multi-Genre Chinese Essay Writingcited by 2

50 most cited of 95 All 95 (JSON)