# Leveraging Open-Source Large Language Models for Clinical Information Extraction in Resource-Constrained Settings

arXiv: https://arxiv.org/abs/2507.20859 · HTML: https://mlsanity.com/paper/2507.20859/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 47 references that are in the map.
- First posted on arXiv 28 Jul 2025.

## Builds on (most cited first, 23 of 23)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Phi-4 Technical Report](https://mlsanity.com/paper/2412.08905/) (2024)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization](https://mlsanity.com/paper/2309.07430/) (2023)
- [Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs](https://mlsanity.com/paper/2402.03927/) (2024)
- [Do NLP Models Know Numbers? Probing Numeracy in Embeddings](https://mlsanity.com/paper/1909.07940/) (2019)
- [MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks](https://mlsanity.com/paper/2311.07463/) (2023)
- [Beyond English: The Impact of Prompt Translation Strategies across Languages and Tasks in Multilingual LLMs](https://mlsanity.com/paper/2502.09331/) (2025)
- [Translation and Fusion Improves Zero-shot Cross-lingual Information Extraction](https://mlsanity.com/paper/2305.13582/) (2023)
- [Breaking the Language Barrier: Can Direct Inference Outperform Pre-Translation in Multilingual LLM Applications?](https://mlsanity.com/paper/2403.04792/) (2024)
- [Language Resources for Dutch Large Language Modelling](https://mlsanity.com/paper/2312.12852/) (2023)
- [Do Large Language Models Speak All Languages Equally? A Comparative Study in Low-Resource Settings](https://mlsanity.com/paper/2408.02237/) (2024)
- [Introducing L2M3, A Multilingual Medical Large Language Model to Advance Health Equity in Low-Resource Regions](https://mlsanity.com/paper/2404.08705/) (2024)
- [Domain-Specific Translation with Open-Source Large Language Models: Resource-Oriented Analysis](https://mlsanity.com/paper/2412.05862/) (2024)

## Built on by (most cited first, 7 of 7)

- [Counterfactual Cultural Cues Reduce Medical QA Accuracy in LLMs: Identifier vs Context Effects](https://mlsanity.com/paper/2601.20102/) (2026)
- [Toward Federated Large Language Models in Medicine: A Parameter-Efficient Framework for Privacy-Preserving, Multi-Institutional Adaptation](https://mlsanity.com/paper/2601.22124/) (2026)
- [Benchmarking and Adapting On-Device LLMs for Clinical Decision Support](https://mlsanity.com/paper/2601.03266/) (2025)
- [Tracking Cancer Through Text: Longitudinal Extraction From Radiology Reports Using Open-Source Large Language Models](https://mlsanity.com/paper/2603.09638/) (2026)
- [Revisiting Gene Ontology Knowledge Discovery with Hierarchical Feature Selection and Virtual Study Group of AI Agents](https://mlsanity.com/paper/2603.20132/) (2026)
- [A Multi-Stage Validation Framework for Trustworthy Large-scale Clinical Information Extraction using Large Language Models](https://mlsanity.com/paper/2604.06028/) (2026)
- [Automatic Extraction of Structured Information from Brain MRI Reports Using an Open-Weight Large Language Model](https://mlsanity.com/paper/2606.07721/) (2026)

## Abstract (arXiv)

Medical reports contain rich clinical information but are often unstructured and written in domain-specific language, posing challenges for information extraction. While proprietary large language models (LLMs) have shown promise in clinical natural language processing, their lack of transparency and data privacy concerns limit their utility in healthcare. This study therefore evaluates nine open-source generative LLMs on the DRAGON benchmark, which includes 28 clinical information extraction tasks in Dutch. We developed textttllmextractinator, a publicly available framework for information extraction using open-source generative LLMs, and used it to assess model performance in a zero-shot setting. Several 14 billion parameter models, Phi-4-14B, Qwen-2.5-14B, and DeepSeek-R1-14B, achieved competitive results, while the bigger Llama-3.3-70B model achieved slightly higher performance at greater computational cost. Translation to English prior to inference consistently degraded performance, highlighting the need of native-language processing. These findings demonstrate that open-source LLMs, when used with our framework, offer effective, scalable, and privacy-conscious solutions for clinical information extraction in low-resource settings.