# BERT Based Clinical Knowledge Extraction for Biomedical Knowledge Graph Construction and Analysis

arXiv: https://arxiv.org/abs/2304.10996 · HTML: https://mlsanity.com/paper/2304.10996/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 52 references that are in the map.
- First posted on arXiv 21 Apr 2023.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [A Survey on Knowledge Graphs: Representation, Acquisition and Applications](https://mlsanity.com/paper/2002.00388/) (2020)
- [Knowledge Graphs](https://mlsanity.com/paper/2003.02320/) (2020)
- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [KGAT: Knowledge Graph Attention Network for Recommendation](https://mlsanity.com/paper/1905.07854/) (2019)
- [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://mlsanity.com/paper/1603.01354/) (2016)
- [BERT for Coreference Resolution: Baselines and Analysis](https://mlsanity.com/paper/1908.09091/) (2019)
- [Unifying Knowledge Graph Learning and Recommendation: Towards a Better Understanding of User Preferences](https://mlsanity.com/paper/1902.06236/) (2019)
- [Enriching Pre-trained Language Model with Entity Information for Relation Classification](https://mlsanity.com/paper/1905.08284/) (2019)
- [Revealing the Myth of Higher-Order Inference in Coreference Resolution](https://mlsanity.com/paper/2009.12013/) (2020)
- [Building a PubMed knowledge graph](https://mlsanity.com/paper/2005.04308/) (2020)
- [An End-to-end Model for Entity-level Relation Extraction using Multi-instance Learning](https://mlsanity.com/paper/2102.05980/) (2021)
- [Drug Repurposing for COVID-19 via Knowledge Graph Completion](https://mlsanity.com/paper/2010.09600/) (2020)
- [Drug-Drug Interaction Prediction Based on Knowledge Graph Embeddings and Convolutional-LSTM Network](https://mlsanity.com/paper/1908.01288/) (2019)
- [Semantic Relation Classification via Bidirectional LSTM Networks with Entity-aware Attention using Latent Entity Typing](https://mlsanity.com/paper/1901.08163/) (2019)

## Built on by (most cited first, 18 of 18)

- [GraphEval: A Lightweight Graph-Based LLM Framework for Idea Evaluation](https://mlsanity.com/paper/2503.12600/) (2025)
- [Inference of Dependency Knowledge Graph for Electronic Health Records](https://mlsanity.com/paper/2312.15611/) (2023)
- [Uncertainty Management in the Construction of Knowledge Graphs: a Survey](https://mlsanity.com/paper/2405.16929/) (2024)
- [Multi-modal Graph Learning over UMLS Knowledge Graphs](https://mlsanity.com/paper/2307.04461/) (2023)
- [Domain Knowledge Graph Construction Via A Simple Checker](https://mlsanity.com/paper/2310.04949/) (2023)
- [Augmenting Biomedical Named Entity Recognition with General-domain Resources](https://mlsanity.com/paper/2406.10671/) (2024)
- [SeLIP: Similarity Enhanced Contrastive Language Image Pretraining for Multi-modal Head MRI](https://mlsanity.com/paper/2503.19801/) (2025)
- [Analysis of the Evolution of Advanced Transformer-Based Language Models: Experiments on Opinion Mining](https://mlsanity.com/paper/2308.03235/) (2023)
- [Learning a Patent-Informed Biomedical Knowledge Graph Reveals Technological Potential of Drug Repositioning Candidates](https://mlsanity.com/paper/2309.03227/) (2023)
- [Enhancing Knowledge Graph Construction: Evaluating with Emphasis on Hallucination, Omission, and Graph Similarity Metrics](https://mlsanity.com/paper/2502.05239/) (2025)
- [DiagLink: A Dual-User Diagnostic Assistance System by Synergizing Experts with LLMs and Knowledge Graphs](https://mlsanity.com/paper/2601.20311/) (2026)
- [MasonNLP+ at SemEval-2023 Task 8: Extracting Medical Questions, Experiences and Claims from Social Media using Knowledge-Augmented Pre-trained Language Models](https://mlsanity.com/paper/2304.13875/) (2023)
- [RECAP-KG: Mining Knowledge Graphs from Raw GP Notes for Remote COVID-19 Assessment in Primary Care](https://mlsanity.com/paper/2306.17175/) (2023)
- [Scalable Knowledge Graph Construction and Inference on Human Genome Variants](https://mlsanity.com/paper/2312.04423/) (2023)
- [A Scalable Tool For Analyzing Genomic Variants Of Humans Using Knowledge Graphs and Machine Learning](https://mlsanity.com/paper/2407.20879/) (2024)
- [RGDA-DDI: Residual graph attention network and dual-attention based framework for drug-drug interaction prediction](https://mlsanity.com/paper/2408.15310/) (2024)
- [A Robust BERT-Based Deep Learning Model for Automated Cancer Type Extraction from Unstructured Pathology Reports](https://mlsanity.com/paper/2508.15149/) (2025)
- [DREaM: Drug-Drug Relation Extraction via Transfer Learning Method](https://mlsanity.com/paper/2510.23189/) (2025)

## Abstract (arXiv)

Background : Knowledge is evolving over time, often as a result of new discoveries or changes in the adopted methods of reasoning. Also, new facts or evidence may become available, leading to new understandings of complex phenomena. This is particularly true in the biomedical field, where scientists and physicians are constantly striving to find new methods of diagnosis, treatment and eventually cure. Knowledge Graphs (KGs) offer a real way of organizing and retrieving the massive and growing amount of biomedical knowledge. Objective : We propose an end-to-end approach for knowledge extraction and analysis from biomedical clinical notes using the Bidirectional Encoder Representations from Transformers (BERT) model and Conditional Random Field (CRF) layer. Methods : The approach is based on knowledge graphs, which can effectively process abstract biomedical concepts such as relationships and interactions between medical entities. Besides offering an intuitive way to visualize these concepts, KGs can solve more complex knowledge retrieval problems by simplifying them into simpler representations or by transforming the problems into representations from different perspectives. We created a biomedical Knowledge Graph using using Natural Language Processing models for named entity recognition and relation extraction. The generated biomedical knowledge graphs (KGs) are then used for question answering. Results : The proposed framework can successfully extract relevant structured information with high accuracy (90.7% for Named-entity recognition (NER), 88% for relation extraction (RE)), according to experimental findings based on real-world 505 patient biomedical unstructured clinical notes. Conclusions : In this paper, we propose a novel end-to-end system for the construction of a biomedical knowledge graph from clinical textual using a variation of BERT models.