# DocReward: A Document Reward Model for Structuring and Stylizing

arXiv: https://arxiv.org/abs/2510.11391 · HTML: https://mlsanity.com/paper/2510.11391/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 28 of its 84 references that are in the map.
- First posted on arXiv 13 Oct 2025.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)
- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 28 of 28)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://mlsanity.com/paper/1912.13318/) (2019)
- [DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning](https://mlsanity.com/paper/2505.14362/) (2025)
- [OmniGen: Unified Image Generation](https://mlsanity.com/paper/2409.11340/) (2024)
- [TravelPlanner: A Benchmark for Real-World Planning with Language Agents](https://mlsanity.com/paper/2402.01622/) (2024)
- [The Impact of AI on Developer Productivity: Evidence from GitHub Copilot](https://mlsanity.com/paper/2302.06590/) (2023)
- [Training-Free Group Relative Policy Optimization](https://mlsanity.com/paper/2510.08191/) (2025)
- [LayoutPrompter: Awaken the Design Ability of Large Language Models](https://mlsanity.com/paper/2311.06495/) (2023)
- [Visual Agentic AI for Spatial Reasoning with a Dynamic API](https://mlsanity.com/paper/2502.06787/) (2025)
- [A Survey of Deep Learning Approaches for OCR and Document Understanding](https://mlsanity.com/paper/2011.13534/) (2020)
- [A-Lamp: Adaptive Layout-Aware Multi-Patch Deep Convolutional Neural Network for Photo Aesthetic Assessment](https://mlsanity.com/paper/1704.00248/) (2017)
- [Towards Aligned Layout Generation via Diffusion Model with Aesthetic Constraints](https://mlsanity.com/paper/2402.04754/) (2024)
- [A Comparative Analysis of Large Language Models for Code Documentation Generation](https://mlsanity.com/paper/2312.10349/) (2023)
- [MathAgent: Leveraging a Mixture-of-Math-Agent Framework for Real-World Multimodal Mathematical Error Detection](https://mlsanity.com/paper/2503.18132/) (2025)
- [LayoutNUWA: Revealing the Hidden Layout Expertise of Large Language Models](https://mlsanity.com/paper/2309.09506/) (2023)
- [AI-Empowered Persuasive Video Generation: A Survey](https://mlsanity.com/paper/2112.09401/) (2021)
- [AesthetiQ: Enhancing Graphic Layout Design via Aesthetic-Aware Preference Alignment of Multi-modal Large Language Models](https://mlsanity.com/paper/2503.00591/) (2025)
- [ReLayout: Towards Real-World Document Understanding via Layout-enhanced Pre-training](https://mlsanity.com/paper/2410.10471/) (2024)
- [NapierOne: A modern mixed file data set alternative to Govdocs1](https://mlsanity.com/paper/2201.08154/) (2022)
- [ReLayout: Integrating Relation Reasoning for Content-aware Layout Generation with Multi-modal Large Language Models](https://mlsanity.com/paper/2507.05568/) (2025)

## Built on by (most cited first, 1 of 1)

- [PaperFit: Vision-in-the-Loop Typesetting Optimization for Scientific Documents](https://mlsanity.com/paper/2605.10341/) (2026)

## Abstract (arXiv)

Recent agentic workflows automate professional document generation but focus narrowly on textual quality, overlooking structural and stylistic professionalism, which is equally critical for readability. This gap stems mainly from a lack of effective reward models capable of guiding agents toward producing documents with high structural and stylistic professionalism. We introduce DocReward, a document reward model that evaluates documents based on their structure and style. To achieve this, we propose a textual-quality-agnostic framework that ensures assessments are not confounded by content quality, and construct DocPair, a dataset of 117K paired documents covering 32 domains and 267 types. Each pair shares identical content but differs in structural and stylistic professionalism. DocReward is trained using the Bradley-Terry loss. On a manually annotated benchmark, DocReward outperforms GPT-5 by 14.6 percentage points in the same setting. Reinforcement learning experiments further show that DocReward effectively guides agents toward generating documents with consistently higher structural and stylistic professionalism, highlighting its practical utility.