# DocPO: Advancing Document Policy Optimization via Tailored Step-Aware Rewards

arXiv: https://arxiv.org/abs/2608.00536 · HTML: https://mlsanity.com/paper/2608.00536/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 33 references that are in the map.
- First posted on arXiv 1 Aug 2026.

## Builds on (most cited first, 19 of 19)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Training Language Models to Self-Correct via Reinforcement Learning](https://mlsanity.com/paper/2409.12917/) (2024)
- [MinerU: An Open-Source Solution for Precise Document Content Extraction](https://mlsanity.com/paper/2409.18839/) (2024)
- [Nougat: Neural Optical Understanding for Academic Documents](https://mlsanity.com/paper/2308.13418/) (2023)
- [MinerU2.5: A Decoupled Vision-Language Model for Efficient High-Resolution Document Parsing](https://mlsanity.com/paper/2509.22186/) (2025)
- [KOSMOS-2.5: A Multimodal Literate Model](https://mlsanity.com/paper/2309.11419/) (2023)
- [MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigm](https://mlsanity.com/paper/2506.05218/) (2025)
- [LayoutReader: Pre-training of Text and Layout for Reading Order Detection](https://mlsanity.com/paper/2108.11591/) (2021)
- [Dolphin: Document Image Parsing via Heterogeneous Anchor Prompting](https://mlsanity.com/paper/2505.14059/) (2025)
- [TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergy](https://mlsanity.com/paper/2406.01326/) (2024)
- [TextSquare: Scaling up Text-Centric Visual Instruction Tuning](https://mlsanity.com/paper/2404.12803/) (2024)
- [A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understanding](https://mlsanity.com/paper/2407.01976/) (2024)
- [Harmonizing Visual Text Comprehension and Generation](https://mlsanity.com/paper/2407.16364/) (2024)
- [Multi-modal In-Context Learning Makes an Ego-evolving Scene Text Recognizer](https://mlsanity.com/paper/2311.13120/) (2023)
- [Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsing](https://mlsanity.com/paper/2510.15349/) (2025)
- [MonkeyOCR v1.5 Technical Report: Unlocking Robust Document Parsing for Complex Patterns](https://mlsanity.com/paper/2511.10390/) (2025)
- [Rethinking Image-based Table Recognition Using Weakly Supervised Methods](https://mlsanity.com/paper/2303.07641/) (2023)
- [StrucTab: A Structured Optimization Framework for Table Parsing](https://mlsanity.com/paper/2606.29905/) (2026)
- [MORE: A Multilingual Document Parsing Benchmark and Evaluation](https://mlsanity.com/paper/2607.02956/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) for document parsing often relies on reference-based rewards rooted in edit distance (e.g., tree edit distance), yet it remains hard to optimize in the high-accuracy regime because such rewards become weakly discriminative: near-correct outputs receive very similar scores, providing limited learning signal for hard cases. We propose Step-Aware Annealing (SAA), a plug-and-play reward sharpening mechanism that progressively increases reward curvature during training, amplifying subtle quality differences among high-scoring samples while preserving stability in early learning. Built on SAA, we introduce DocPO, a document policy optimization framework with element-specific, reference-based rewards anchored by edit-distance signals: normalized string edit distance (NED) for text, tree edit distance similarity (TEDS) for tables, and a hybrid Rubric+edit reward for formulas. Experiments on OmniDocBench and DocElemHard show that SAA consistently improves GRPO-style RL across document elements over non-annealed rewards, without requiring additional human supervision for reward construction.