# Improving Document Image Understanding with Reinforcement Finetuning

arXiv: https://arxiv.org/abs/2209.12561 · HTML: https://mlsanity.com/paper/2209.12561/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 25 references that are in the map.
- First posted on arXiv 26 Sep 2022.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://mlsanity.com/paper/1912.13318/) (2019)
- [LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding](https://mlsanity.com/paper/2012.14740/) (2020)
- [Deep Reinforcement Learning for Dialogue Generation](https://mlsanity.com/paper/1606.01541/) (2016)
- [ICDAR2019 Competition on Scanned Receipt OCR and Information Extraction](https://mlsanity.com/paper/2103.10213/) (2021)
- [Graph Convolution for Multimodal Information Extraction from Visually Rich Documents](https://mlsanity.com/paper/1903.11279/) (2019)
- [A Study of Reinforcement Learning for Neural Machine Translation](https://mlsanity.com/paper/1808.08866/) (2018)
- [PICK: Processing Key Information Extraction from Documents using Improved Graph Learning-Convolutional Networks](https://mlsanity.com/paper/2004.07464/) (2020)
- [Deep Communicating Agents for Abstractive Summarization](https://mlsanity.com/paper/1803.10357/) (2018)
- [A Span-Based Model for Joint Overlapped and Discontinuous Named Entity Recognition](https://mlsanity.com/paper/2106.14373/) (2021)
- [Visual FUDGE: Form Understanding via Dynamic Graph Editing](https://mlsanity.com/paper/2105.08194/) (2021)
- [Make The Most of Prior Data: A Solution for Interactive Text Summarization with Preference Feedback](https://mlsanity.com/paper/2204.05512/) (2022)
- [Robust Deep Reinforcement Learning for Extractive Legal Summarization](https://mlsanity.com/paper/2111.07158/) (2021)
- [Towards Few-shot Entity Recognition in Document Images: A Label-aware Sequence-to-Sequence Framework](https://mlsanity.com/paper/2204.05819/) (2022)
- [A Span Extraction Approach for Information Extraction on Visually-Rich Documents](https://mlsanity.com/paper/2106.00978/) (2021)
- [Episodic Policy Gradient Training](https://mlsanity.com/paper/2112.01853/) (2021)
- [Jointly Learning Span Extraction and Sequence Labeling for Information Extraction from Business Documents](https://mlsanity.com/paper/2205.13434/) (2022)

## Abstract (arXiv)

Successful Artificial Intelligence systems often require numerous labeled data to extract information from document images. In this paper, we investigate the problem of improving the performance of Artificial Intelligence systems in understanding document images, especially in cases where training data is limited. We address the problem by proposing a novel finetuning method using reinforcement learning. Our approach treats the Information Extraction model as a policy network and uses policy gradient training to update the model to maximize combined reward functions that complement the traditional cross-entropy losses. Our experiments on four datasets using labels and expert feedback demonstrate that our finetuning mechanism consistently improves the performance of a state-of-the-art information extractor, especially in the small training data regime.