# From Coarse to Fine: Benchmarking and Reward Modeling for Writing-Centric Generation Tasks

arXiv: https://arxiv.org/abs/2604.27453 · HTML: https://mlsanity.com/paper/2604.27453/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 28 of its 40 references that are in the map.
- First posted on arXiv 30 Apr 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 28 of 28)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [RewardBench: Evaluating Reward Models for Language Modeling](https://mlsanity.com/paper/2403.13787/) (2024)
- [LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset](https://mlsanity.com/paper/2309.11998/) (2023)
- [JudgeBench: A Benchmark for Evaluating LLM-based Judges](https://mlsanity.com/paper/2410.12784/) (2024)
- [DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents](https://mlsanity.com/paper/2506.11763/) (2025)
- [The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models](https://mlsanity.com/paper/2404.16019/) (2024)
- [LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs](https://mlsanity.com/paper/2408.07055/) (2024)
- [WritingBench: A Comprehensive Benchmark for Generative Writing](https://mlsanity.com/paper/2503.05244/) (2025)
- [Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewards](https://mlsanity.com/paper/2506.00103/) (2025)
- [VerIF: Verification Engineering for Reinforcement Learning in Instruction Following](https://mlsanity.com/paper/2506.09942/) (2025)
- [Suri: Multi-constraint Instruction Following for Long-form Text Generation](https://mlsanity.com/paper/2406.19371/) (2024)
- [CollabStory: Multi-LLM Collaborative Story Generation and Authorship Analysis](https://mlsanity.com/paper/2406.12665/) (2024)
- [Language Models can Self-Lengthen to Generate Long Texts](https://mlsanity.com/paper/2410.23933/) (2024)
- [LLM-RG4: Flexible and Factual Radiology Report Generation across Diverse Input Contexts](https://mlsanity.com/paper/2412.12001/) (2024)
- [RLMR: Reinforcement Learning with Mixed Rewards for Creative Writing](https://mlsanity.com/paper/2508.18642/) (2025)
- [Igniting Creative Writing in Small Language Models: LLM-as-a-Judge versus Multi-Agent Refined Rewards](https://mlsanity.com/paper/2508.21476/) (2025)
- [How Far Are We from Genuinely Useful Deep Research Agents?](https://mlsanity.com/paper/2512.01948/) (2025)
- [ACE-RL: Adaptive Constraint-Enhanced Reward for Long-form Generation Reinforcement Learning](https://mlsanity.com/paper/2509.04903/) (2025)
- [FashionReGen: LLM-Empowered Fashion Report Generation](https://mlsanity.com/paper/2403.06660/) (2024)
- [Adaptive LLM-Symbolic Reasoning via Dynamic Logical Solver Composition](https://mlsanity.com/paper/2510.06774/) (2025)
- [Instructions are all you need: Self-supervised Reinforcement Learning for Instruction Following](https://mlsanity.com/paper/2510.14420/) (2025)
- [LLM Review: Enhancing Creative Writing via Blind Peer Review Feedback](https://mlsanity.com/paper/2601.08003/) (2026)
- [Context-Enhanced Contrastive Search for Improved LLM Text Generation](https://mlsanity.com/paper/2504.21020/) (2025)
- [Long-form RewardBench: Evaluating Reward Models for Long-form Generation](https://mlsanity.com/paper/2603.12963/) (2026)

## Abstract (arXiv)

Large language models have achieved remarkable progress in text generation but still struggle with generative writing tasks. In terms of evaluation, existing benchmarks evaluate writing reward models coarsely and fail to measure performance from the perspective of specific requirements. In terms of training, existing training methods either use LLM-as-a-judge approaches or train coarse-grained reward models, lacking fine-grained requirement-adherence reward modeling. To address these issues, we propose a fine-grained evaluation pipeline WEval for writing reward models and a fine-grained reinforcement learning training framework WRL. The evaluation data of WEval covers multiple task categories and requirement types, enabling systematic evaluation of writing reward models by measuring the correlation between the rankings of the reward model and gold rankings. WRL constructs positive and negative samples by selectively dropping instruction requirements, allowing for more precise reward model training. Experiments show that our models achieve substantial improvements across various writing benchmarks and exhibit strong generalization. The code and data are publicly available at hrefhttps://github.com/Rainier-rq1/FromCoarsetoFinehttps://github.com/Rainier-rq1/FromCoarsetoFine.