# Efficient Medical VIE via Reinforcement Learning

arXiv: https://arxiv.org/abs/2506.13363 · HTML: https://mlsanity.com/paper/2506.13363/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL for clinical treatment and surgery, one of 15 directions in Deep reinforcement learning methods.
- Builds on 26 of its 44 references that are in the map.
- First posted on arXiv 16 Jun 2025.

## Placed in

- Deep reinforcement learning methods / [RL for clinical treatment and surgery](https://mlsanity.com/deeprl/rl-for-clinical-treatment-and-surgery/)

## Builds on (most cited first, 26 of 26)

- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://mlsanity.com/paper/1912.13318/) (2019)
- [Visual-RFT: Visual Reinforcement Fine-Tuning](https://mlsanity.com/paper/2503.01785/) (2025)
- [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://mlsanity.com/paper/2204.08387/) (2022)
- [LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding](https://mlsanity.com/paper/2012.14740/) (2020)
- [FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents](https://mlsanity.com/paper/1905.13538/) (2019)
- [MinerU: An Open-Source Solution for Precise Document Content Extraction](https://mlsanity.com/paper/2409.18839/) (2024)
- [ICDAR2019 Competition on Scanned Receipt OCR and Information Extraction](https://mlsanity.com/paper/2103.10213/) (2021)
- [DocFormer: End-to-End Transformer for Document Understanding](https://mlsanity.com/paper/2106.11539/) (2021)
- [Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcement](https://mlsanity.com/paper/2503.06520/) (2025)
- [Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformer](https://mlsanity.com/paper/2102.09550/) (2021)
- [olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models](https://mlsanity.com/paper/2502.18443/) (2025)
- [BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documents](https://mlsanity.com/paper/2108.04539/) (2021)
- [DocLayout-YOLO: Enhancing Document Layout Analysis through Diverse Synthetic Data and Global-to-Local Adaptive Perception](https://mlsanity.com/paper/2410.12628/) (2024)
- [OmniParser: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognition](https://mlsanity.com/paper/2403.19128/) (2024)
- [PICK: Processing Key Information Extraction from Documents using Improved Graph Learning-Convolutional Networks](https://mlsanity.com/paper/2004.07464/) (2020)
- [Towards Robust Visual Information Extraction in Real World: New Dataset and Novel Solution](https://mlsanity.com/paper/2102.06732/) (2021)
- [TRIE: End-to-End Text Reading and Information Extraction for Document Understanding](https://mlsanity.com/paper/2005.13118/) (2020)
- [Visual Information Extraction in the Wild: Practical Dataset and End-to-end Solution](https://mlsanity.com/paper/2305.07498/) (2023)
- [GoT-R1: Unleashing Reasoning Capability of MLLM for Visual Generation with Reinforcement Learning](https://mlsanity.com/paper/2505.17022/) (2025)
- [PGNet: Real-time Arbitrarily-Shaped Text Spotting with Point Gathering Network](https://mlsanity.com/paper/2104.05458/) (2021)
- [GeoLayoutLM: Geometric Pre-training for Visual Information Extraction](https://mlsanity.com/paper/2304.10759/) (2023)
- [EATEN: Entity-aware Attention for Single Shot Visual Text Extraction](https://mlsanity.com/paper/1909.09380/) (2019)
- [XDoc: Unified Pre-training for Cross-Format Document Understanding](https://mlsanity.com/paper/2210.02849/) (2022)
- [Exploring the Inquiry-Diagnosis Relationship with Advanced Patient Simulators](https://mlsanity.com/paper/2501.09484/) (2025)
- [A LayoutLMv3-Based Model for Enhanced Relation Extraction in Visually-Rich Documents](https://mlsanity.com/paper/2404.10848/) (2024)

## Built on by (most cited first, 2 of 2)

- [A Survey of Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2509.08827/) (2025)
- [CheXPO-v2: Preference Optimization for Chest X-ray VLMs with Knowledge Graph Consistency](https://mlsanity.com/paper/2512.17213/) (2025)

## Abstract (arXiv)

Visual Information Extraction (VIE) converts unstructured document images into structured formats like JSON, critical for medical applications such as report analysis and online consultations. Traditional methods rely on OCR and language models, while end-to-end multimodal models offer direct JSON generation. However, domain-specific schemas and high annotation costs limit their effectiveness in medical VIE. We base our approach on the Reinforcement Learning with Verifiable Rewards (RLVR) framework to address these challenges using only 100 annotated samples. Our approach ensures dataset diversity, a balanced precision-recall reward mechanism to reduce hallucinations and improve field coverage, and innovative sampling strategies to enhance reasoning capabilities. Fine-tuning Qwen2.5-VL-7B with our RLVR method, we achieve state-of-the-art performance on medical VIE tasks, significantly improving F1, precision, and recall. While our models excel on tasks similar to medical datasets, performance drops on dissimilar tasks, highlighting the need for domain-specific optimization. Case studies further demonstrate the value of reasoning during training and inference for VIE.