mlsanityData to 14 Oct 2026

A Multimodal Pipeline for Clinical Data Extraction: Applying Vision-Language Models to Scans of Transfusion Reaction Reports

arXiv 2504.20220 ↗First posted 28 Apr 2025cs.CL cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Despite the growing adoption of electronic health records, many processes still rely on paper documents, reflecting the heterogeneous real-world conditions in which healthcare is delivered. The manual transcription process is time-consuming and prone to errors when transferring paper-based data to digital formats. To streamline this workflow, this study presents an open-source pipeline that extracts and categorizes checkbox data from scanned documents. Demonstrated on transfusion reaction reports, the design supports adaptation to other checkbox-rich document types. The proposed method integrates checkbox detection, multilingual optical character recognition (OCR) and multilingual vision-language models (VLMs). The pipeline achieves high precision and recall compared against annually compiled gold-standards from 2017 to 2024. The result is a reduction in administrative workload and accurate regulatory reporting. The open-source availability of this pipeline encourages self-hosted parsing of checkbox forms.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 14 of its 33 references that are in the map. First posted on arXiv 28 Apr 2025.

Builds on

14 of 33 references.

References · 14

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  3. 2020End-to-End Object Detection with Transformerscited by 7,414
  4. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  5. 2015VQA: Visual Question Answeringcited by 2,506
  6. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  7. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  8. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 553
  9. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 408
  10. 2021OCR-free Document Understanding Transformercited by 407
  11. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 349
  12. 2015Evaluation of Deep Convolutional Nets for Document Image Classification and Retrievalcited by 213
2 more
  1. 2024Pixtral 12Bcited by 179
  2. 2021PP-OCRv2: Bag of Tricks for Ultra Lightweight OCR Systemcited by 39

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025From Chaos to Clarity: Schema-Constrained AI for Auditable Biomedical Evidence Extraction from Full-Text PDFscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024ChatSchema: A pipeline of extracting structured information with Large Multimodal Models based on schema
  2. 2025Compact Multimodal Language Models as Robust OCR Alternatives for Noisy Textual Clinical Reports ↗
  3. 2026A Hybrid Architecture for Multi-Stage Claim Document Understanding: Combining Vision-Language Models and Machine Learning for Real-Time Processing
  4. 2026ClinOCR-Bench: A Comprehensive Clinical Scanned Document Dataset for Optical Character Recognition Model Evaluation
  5. 2025Analysis of Blood Report Images Using General Purpose Vision-Language Models
  6. 2024From Text to Multimodality: Exploring the Evolution and Impact of Large Language Models in Medical Practice
  7. 2026MeDocVL: A Visual Language Model for Medical Document Understanding and Parsing
  8. 2026Medical Context Distorts Decisions in Clinical Vision Language Models