# Conformal Prediction for Risk-Controlled Medical Entity Extraction Across Clinical Domains

arXiv: https://arxiv.org/abs/2603.00924 · HTML: https://mlsanity.com/paper/2603.00924/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 21 references that are in the map.
- First posted on arXiv 1 Mar 2026.

## Builds on (most cited first, 11 of 11)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [RadGraph: Extracting Clinical Entities and Relations from Radiology Reports](https://mlsanity.com/paper/2106.14463/) (2021)
- [Distribution-Free, Risk-Controlling Prediction Sets](https://mlsanity.com/paper/2101.02703/) (2021)
- [A Survey of Confidence Estimation and Calibration in Large Language Models](https://mlsanity.com/paper/2311.08298/) (2023)
- [Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experiment](https://mlsanity.com/paper/2311.08596/) (2023)
- [Conformal Prediction for Natural Language Processing: A Survey](https://mlsanity.com/paper/2405.01976/) (2024)
- [VeriFact: Verifying Facts in LLM-Generated Clinical Text with Electronic Health Records](https://mlsanity.com/paper/2501.16672/) (2025)
- [Empirical Validation of Conformal Prediction for Trustworthy Skin Lesions Classification](https://mlsanity.com/paper/2312.07460/) (2023)
- [Structured Extraction of Real World Medical Knowledge using LLMs for Summarization and Search](https://mlsanity.com/paper/2412.15256/) (2024)
- [Mining Unstructured Medical Texts With Conformal Active Learning](https://mlsanity.com/paper/2502.04372/) (2025)

## Built on by (most cited first, 1 of 1)

- [CARE: A Conformal Safety Layer for Medical Summarization](https://mlsanity.com/paper/2606.08969/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly used for medical entity extraction, yet their confidence scores are often miscalibrated, limiting safe deployment in clinical settings. We present a conformal prediction framework that provides finite-sample coverage guarantees for LLM-based extraction across two clinical domains. First, we extract structured entities from 1,000 FDA drug labels across eight sections using GPT-4.1, verified via FactScore-based atomic statement evaluation (97.7% accuracy over 128,906 entities). Second, we extract radiological entities from MIMIC-CXR reports using the RadGraph schema with GPT-4.1 and Llama-4-Maverick, evaluated against physician annotations (entity F1: 0.81 to 0.84). Our central finding is that miscalibration direction reverses across domains: on well-structured FDA labels, models are underconfident, requiring modest conformal thresholds (τ ≈ 0.06), while on free-text radiology reports, models are overconfident, demanding strict thresholds (τ up to 0.99). Despite this heterogeneity, conformal prediction achieves target coverage (≥ 90%) in both settings with manageable rejection rates (9--13%). These results demonstrate that calibration is not a global model property but depends on document structure, extraction category, and model architecture, motivating domain-specific conformal calibration for safe clinical deployment.