# Large Language Models for Biomedical Knowledge Graph Construction: Information extraction from EMR notes

arXiv: https://arxiv.org/abs/2301.12473 · HTML: https://mlsanity.com/paper/2301.12473/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 63 references that are in the map.
- First posted on arXiv 29 Jan 2023.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining](https://mlsanity.com/paper/2210.10341/) (2022)
- [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://mlsanity.com/paper/2108.08877/) (2021)
- [UL2: Unifying Language Learning Paradigms](https://mlsanity.com/paper/2205.05131/) (2022)
- [OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization](https://mlsanity.com/paper/2212.12017/) (2022)
- [Large Language Models are Few-Shot Clinical Information Extractors](https://mlsanity.com/paper/2205.12689/) (2022)
- [Can Language Models be Biomedical Knowledge Bases?](https://mlsanity.com/paper/2109.07154/) (2021)
- [Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again](https://mlsanity.com/paper/2203.08410/) (2022)
- [Developing a general-purpose clinical language inference model from a large corpus of clinical notes](https://mlsanity.com/paper/2210.06566/) (2022)
- [Zero-Shot Open-Book Question Answering](https://mlsanity.com/paper/2111.11520/) (2021)

## Built on by (most cited first, 18 of 18)

- [KGGen: Extracting Knowledge Graphs from Plain Text with Language Models](https://mlsanity.com/paper/2502.09956/) (2025)
- [Prompt engineering paradigms for medical applications: scoping review and recommendations for better practices](https://mlsanity.com/paper/2405.01249/) (2024)
- [DKEC: Domain Knowledge Enhanced Multi-Label Classification for Diagnosis Prediction](https://mlsanity.com/paper/2310.07059/) (2023)
- [FactEHR: A Dataset for Evaluating Factuality in Clinical Notes Using LLMs](https://mlsanity.com/paper/2412.12422/) (2024)
- [Foundation Model Sherpas: Guiding Foundation Models through Knowledge and Reasoning](https://mlsanity.com/paper/2402.01602/) (2024)
- [Structured Extraction of Real World Medical Knowledge using LLMs for Summarization and Search](https://mlsanity.com/paper/2412.15256/) (2024)
- [Large Language Models for Knowledge Graph Embedding: A Survey](https://mlsanity.com/paper/2501.07766/) (2025)
- [ConTextual: Improving Clinical Text Summarization in LLMs with Context-preserving Token Filtering and Knowledge Graphs](https://mlsanity.com/paper/2504.16394/) (2025)
- [Reviewing Clinical Knowledge in Medical Large Language Models: Training and Beyond](https://mlsanity.com/paper/2502.20988/) (2025)
- [Zero-Shot Open-Schema Entity Structure Discovery](https://mlsanity.com/paper/2506.04458/) (2025)
- [MEDAKA: Construction of Biomedical Knowledge Graphs Using Large Language Models](https://mlsanity.com/paper/2509.26128/) (2025)
- [StructSynth: Dependency Graphs as Generation Plans for Low-Data Tabular Synthesis with Language Models](https://mlsanity.com/paper/2508.02601/) (2025)
- [TextMineX: Data, Evaluation Framework and Ontology-guided LLM Pipeline for Humanitarian Mine Action](https://mlsanity.com/paper/2509.15098/) (2025)
- [Guide: Generalized-Prior and Data Encoders for DAG Estimation](https://mlsanity.com/paper/2509.23992/) (2025)
- [BioPIE: A Biomedical Protocol Information Extraction Dataset for Experiment Understanding](https://mlsanity.com/paper/2601.04524/) (2026)
- [What Matters When Building Universal Multilingual Named Entity Recognition Models?](https://mlsanity.com/paper/2601.06347/) (2026)
- [Hyper-KGGen: A Skill-Driven Knowledge Extractor for High-Quality Knowledge Hypergraph Generation](https://mlsanity.com/paper/2602.19543/) (2026)
- [Ontology-Based Post-Coordinated Expression Assembly: from Clinical Narrative to Queryable Knowledge Graphs](https://mlsanity.com/paper/2610.16474/) (2026)

## Abstract (arXiv)

The automatic construction of knowledge graphs (KGs) is an important research area in medicine, with far-reaching applications spanning drug discovery and clinical trial design. These applications hinge on the accurate identification of interactions among medical and biological entities. In this study, we propose an end-to-end machine learning solution based on large language models (LLMs) that utilize electronic medical record notes to construct KGs. The entities used in the KG construction process are diseases, factors, treatments, as well as manifestations that coexist with the patient while experiencing the disease. Given the critical need for high-quality performance in medical applications, we embark on a comprehensive assessment of 12 LLMs of various architectures, evaluating their performance and safety attributes. To gauge the quantitative efficacy of our approach by assessing both precision and recall, we manually annotate a dataset provided by the Macula and Retina Institute. We also assess the qualitative performance of LLMs, such as the ability to generate structured outputs or the tendency to hallucinate. The results illustrate that in contrast to encoder-only and encoder-decoder, decoder-only LLMs require further investigation. Additionally, we provide guided prompt design to utilize such LLMs. The application of the proposed methodology is demonstrated on age-related macular degeneration.