# MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports

arXiv: https://arxiv.org/abs/2605.03103 · HTML: https://mlsanity.com/paper/2605.03103/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 28 references that are in the map.
- First posted on arXiv 4 May 2026.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents](https://mlsanity.com/paper/1905.13538/) (2019)
- [RadGraph: Extracting Clinical Entities and Relations from Radiology Reports](https://mlsanity.com/paper/2106.14463/) (2021)
- [Revisiting Pre-Trained Models for Chinese Natural Language Processing](https://mlsanity.com/paper/2004.13922/) (2020)
- [Unified Structure Generation for Universal Information Extraction](https://mlsanity.com/paper/2203.12277/) (2022)
- [InstructUIE: Multi-task Instruction Tuning for Unified Information Extraction](https://mlsanity.com/paper/2304.08085/) (2023)
- [OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations](https://mlsanity.com/paper/2412.07626/) (2024)
- [CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2106.08087/) (2021)
- [Multilingual LLMs are Better Cross-lingual In-context Learners with Alignment](https://mlsanity.com/paper/2305.05940/) (2023)
- [Baichuan-M2: Scaling Medical Capability with Large Verifier System](https://mlsanity.com/paper/2509.02208/) (2025)
- [A Benchmark for Automatic Medical Consultation System: Frameworks, Tasks and Datasets](https://mlsanity.com/paper/2204.08997/) (2022)
- [PromptCBLUE: A Chinese Prompt Tuning Benchmark for the Medical Domain](https://mlsanity.com/paper/2310.14151/) (2023)
- [Docopilot: Improving Multimodal Models for Document-Level Understanding](https://mlsanity.com/paper/2507.14675/) (2025)
- [MC-BERT: Efficient Language Pre-Training via a Meta Controller](https://mlsanity.com/paper/2006.05744/) (2020)
- [Information Extraction from Visually Rich Documents using LLM-based Organization of Documents into Independent Textual Segments](https://mlsanity.com/paper/2505.13535/) (2025)
- [EHRStruct: A Comprehensive Benchmark Framework for Evaluating Large Language Models on Structured Electronic Health Record Tasks](https://mlsanity.com/paper/2511.08206/) (2025)

## Built on by (most cited first, 1 of 1)

- [Key Coverage Matters: Semi-Structured Extraction of OCR Clinical Reports](https://mlsanity.com/paper/2605.09440/) (2026)

## Abstract (arXiv)

Semi-structured information extraction (IE) from OCR-derived clinical reports is crucial for efficiently reconstructing patients' longitudinal medical histories. In practice, this scenario commonly involves three tasks: (i) field-header (key) discovery, (ii) key-conditioned question answering (QA), and (iii) end-to-end key-value pair extraction. However, existing evaluations often under-model two factors: heterogeneous and incompletely known key representations, and OCR-induced noise. This makes it difficult to assess model robustness in real-world settings. We present MedStruct-S, a benchmark specifically designed to evaluate these tasks under unknown keys and OCR noise. MedStruct-S contains 3,582 annotated real-world clinical report pages. Using MedStruct-S, we benchmark two representative paradigms: encoder-only sequence labeling with post-processing and decoder-only structured generation, covering four encoder-only and five decoder-only models spanning 0.11B to 103B parameters. Our results show that encoder-only models achieve the best performance for non-null-value key-conditioned QA despite being substantially smaller than decoder-only models. When comparing models of similar order of magnitude, encoder-only models still perform better overall. Without controlling for model scale, fine-tuned decoder-only models deliver the strongest overall results. These findings show that the benchmark provides a reliable and practical basis for selecting and comparing models across different semi-structured IE settings.