# Leveraging Large Language Models for Knowledge-free Weak Supervision in Clinical Natural Language Processing

arXiv: https://arxiv.org/abs/2406.06723 · HTML: https://mlsanity.com/paper/2406.06723/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 32 references that are in the map.
- First posted on arXiv 10 Jun 2024.

## Builds on (most cited first, 12 of 12)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Large Language Models are Few-Shot Clinical Information Extractors](https://mlsanity.com/paper/2205.12689/) (2022)
- [Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again](https://mlsanity.com/paper/2203.08410/) (2022)
- [A Survey on Programmatic Weak Supervision](https://mlsanity.com/paper/2202.05433/) (2022)
- [Language Models in the Loop: Incorporating Prompting into Weak Supervision](https://mlsanity.com/paper/2205.02318/) (2022)
- [GPT-3 Models are Poor Few-Shot Learners in the Biomedical Domain](https://mlsanity.com/paper/2109.02555/) (2021)
- [Ontology-driven weak supervision for clinical entity classification in electronic health records](https://mlsanity.com/paper/2008.01972/) (2020)
- [Ontology-Driven and Weakly Supervised Rare Disease Identification from Clinical Notes](https://mlsanity.com/paper/2205.05656/) (2022)

## Built on by (most cited first, 8 of 8)

- [LLM-IE: A Python Package for Generative Information Extraction with Large Language Models](https://mlsanity.com/paper/2411.11779/) (2024)
- [SynthEHR-Eviction: Enhancing Eviction SDoH Detection with LLM-Augmented Synthetic EHR Data](https://mlsanity.com/paper/2507.07421/) (2025)
- [CancerGUIDE: Cancer Guideline Understanding via Internal Disagreement Estimation](https://mlsanity.com/paper/2509.07325/) (2025)
- [Prior-informed optimization of treatment recommendation via bandit algorithms trained on large language model-processed historical records](https://mlsanity.com/paper/2510.19014/) (2025)
- [Synthesized Annotation Guidelines are Knowledge-Lite Boosters for Clinical Information Extraction](https://mlsanity.com/paper/2504.02871/) (2025)
- [MechAInistic: An LLM-guided Multi-Agent System for Reasoning over Genome-Scale Constraint-Based Metabolic Models](https://mlsanity.com/paper/2607.18249/) (2026)
- [Large language models as synthetic clinical experts to inform longitudinal rare-disease modeling](https://mlsanity.com/paper/2608.16507/) (2026)
- [Ontology-Based Post-Coordinated Expression Assembly: from Clinical Narrative to Queryable Knowledge Graphs](https://mlsanity.com/paper/2610.16474/) (2026)

## Abstract (arXiv)

The performance of deep learning-based natural language processing systems is based on large amounts of labeled training data which, in the clinical domain, are not easily available or affordable. Weak supervision and in-context learning offer partial solutions to this issue, particularly using large language models (LLMs), but their performance still trails traditional supervised methods with moderate amounts of gold-standard data. In particular, inferencing with LLMs is computationally heavy. We propose an approach leveraging fine-tuning LLMs and weak supervision with virtually no domain knowledge that still achieves consistently dominant performance. Using a prompt-based approach, the LLM is used to generate weakly-labeled data for training a downstream BERT model. The weakly supervised model is then further fine-tuned on small amounts of gold standard data. We evaluate this approach using Llama2 on three different n2c2 datasets. With no more than 10 gold standard notes, our final BERT models weakly supervised by fine-tuned Llama2-13B consistently outperformed out-of-the-box PubMedBERT by 4.7% to 47.9% in F1 scores. With only 50 gold standard notes, our models achieved close performance to fully fine-tuned systems.