# WritingBench: A Comprehensive Benchmark for Generative Writing

arXiv: https://arxiv.org/abs/2503.05244 · HTML: https://mlsanity.com/paper/2503.05244/

## Key facts

- Cited by 106 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 31 references that are in the map.
- First posted on arXiv 7 Mar 2025.

## Builds on (most cited first, 20 of 20)

- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline](https://mlsanity.com/paper/2406.11939/) (2024)
- [Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionals](https://mlsanity.com/paper/2209.14958/) (2022)
- [Large Language Models for Education: A Survey and Outlook](https://mlsanity.com/paper/2403.18105/) (2024)
- [Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models](https://mlsanity.com/paper/2402.14207/) (2024)
- [LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs](https://mlsanity.com/paper/2408.07055/) (2024)
- [A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writing](https://mlsanity.com/paper/2310.08433/) (2023)
- [EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models](https://mlsanity.com/paper/2312.06281/) (2023)
- [One Thousand and One Pairs: A "novel" challenge for long-context language models](https://mlsanity.com/paper/2406.16264/) (2024)
- [SuperCLUE: A Comprehensive Chinese Large Language Model Benchmark](https://mlsanity.com/paper/2307.15020/) (2023)
- [HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models](https://mlsanity.com/paper/2409.16191/) (2024)
- [Suri: Multi-constraint Instruction Following for Long-form Text Generation](https://mlsanity.com/paper/2406.19371/) (2024)
- [Weaver: Foundation Models for Creative Writing](https://mlsanity.com/paper/2401.17268/) (2024)
- [Pron vs Prompt: Can Large Language Models already Challenge a World-Class Fiction Author at Creative Text Writing?](https://mlsanity.com/paper/2407.01119/) (2024)
- [Adapting Large Language Models for Education: Foundational Capabilities, Potentials, and Challenges](https://mlsanity.com/paper/2401.08664/) (2023)
- [MoPS: Modular Story Premise Synthesis for Open-Ended Automatic Story Generation](https://mlsanity.com/paper/2406.05690/) (2024)
- [A Sentiment Consolidation Framework for Meta-Review Generation](https://mlsanity.com/paper/2402.18005/) (2024)
- [Ex3: Automatic Novel Writing by Extracting, Excelsior and Expanding](https://mlsanity.com/paper/2408.08506/) (2024)
- [Fennec: Fine-grained Language Model Evaluation and Correction Extended through Branching and Bridging](https://mlsanity.com/paper/2405.12163/) (2024)

## Built on by (most cited first, 30 of 106)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Qwen3-Omni Technical Report](https://mlsanity.com/paper/2509.17765/) (2025)
- [DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents](https://mlsanity.com/paper/2506.11763/) (2025)
- [Reinforcement Learning with Rubric Anchors](https://mlsanity.com/paper/2508.12790/) (2025)
- [Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoning](https://mlsanity.com/paper/2508.16949/) (2025)
- [MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systems](https://mlsanity.com/paper/2510.17281/) (2025)
- [Baichuan-M2: Scaling Medical Capability with Large Verifier System](https://mlsanity.com/paper/2509.02208/) (2025)
- [Expanding the Capabilities of Reinforcement Learning via Text Feedback](https://mlsanity.com/paper/2602.02482/) (2026)
- [Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewards](https://mlsanity.com/paper/2506.00103/) (2025)
- [Reverse-Engineered Reasoning for Open-Ended Generation](https://mlsanity.com/paper/2509.06160/) (2025)
- [Reward Hacking in Rubric-Based Reinforcement Learning](https://mlsanity.com/paper/2605.12474/) (2026)
- [DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle](https://mlsanity.com/paper/2512.04324/) (2025)
- [Evaluating LLM Metrics Through Real-World Capabilities](https://mlsanity.com/paper/2505.08253/) (2025)
- [ArenaRL: Scaling RL for Open-Ended Agents via Tournament-based Relative Ranking](https://mlsanity.com/paper/2601.06487/) (2026)
- [Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learning](https://mlsanity.com/paper/2606.04923/) (2026)
- [Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifiers](https://mlsanity.com/paper/2606.12507/) (2026)
- [Reinforcement Learning Foundations for Deep Research Systems: A Survey](https://mlsanity.com/paper/2509.06733/) (2025)
- [LongWriter-Zero: Mastering Ultra-Long Text Generation via Reinforcement Learning](https://mlsanity.com/paper/2506.18841/) (2025)
- [EvalAgent: Discovering Implicit Evaluation Criteria from the Web](https://mlsanity.com/paper/2504.15219/) (2025)
- [RLMR: Reinforcement Learning with Mixed Rewards for Creative Writing](https://mlsanity.com/paper/2508.18642/) (2025)
- [A Survey of AI Scientists](https://mlsanity.com/paper/2510.23045/) (2025)
- [EvoRubric: Self-Evolving Rubric-Driven RL for Open-Ended Generation](https://mlsanity.com/paper/2605.29847/) (2026)
- [Evaluating Text Creativity across Diverse Domains: A Dataset and Large Language Model Evaluator](https://mlsanity.com/paper/2505.19236/) (2025)
- [Attack via Overfitting: 10-shot Benign Fine-tuning to Jailbreak LLMs](https://mlsanity.com/paper/2510.02833/) (2025)
- [SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding](https://mlsanity.com/paper/2604.09557/) (2026)
- [ACE-RL: Adaptive Constraint-Enhanced Reward for Long-form Generation Reinforcement Learning](https://mlsanity.com/paper/2509.04903/) (2025)
- [Understanding DeepResearch via Reports](https://mlsanity.com/paper/2510.07861/) (2025)
- [The Personalization Trap: How User Memory Alters Emotional Reasoning in LLMs](https://mlsanity.com/paper/2510.09905/) (2025)
- [Instructions are all you need: Self-supervised Reinforcement Learning for Instruction Following](https://mlsanity.com/paper/2510.14420/) (2025)

## Abstract (arXiv)

Recent advancements in large language models (LLMs) have significantly enhanced text generation capabilities, yet evaluating their performance in generative writing remains a challenge. Existing benchmarks primarily focus on generic text generation or limited in writing tasks, failing to capture the diverse requirements of high-quality written contents across various domains. To bridge this gap, we present WritingBench, a comprehensive benchmark designed to evaluate LLMs across 6 core writing domains and 100 subdomains. We further propose a query-dependent evaluation framework that empowers LLMs to dynamically generate instance-specific assessment criteria. This framework is complemented by a fine-tuned critic model for criteria-aware scoring, enabling evaluations in style, format and length. The framework's validity is further demonstrated by its data curation capability, which enables a 7B-parameter model to outperform the performance of GPT-4o in writing. We open-source the benchmark, along with evaluation tools and modular framework components, to advance the development of LLMs in writing.