# Neural Natural Language Processing for Unstructured Data in Electronic Health Records: a Review

arXiv: https://arxiv.org/abs/2107.02975 · HTML: https://mlsanity.com/paper/2107.02975/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 138 of its 301 references that are in the map.
- First posted on arXiv 7 Jul 2021.

## Builds on (most cited first, 30 of 138)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission](https://mlsanity.com/paper/1904.05342/) (2019)
- [PathVQA: 30000+ Questions for Medical Visual Question Answering](https://mlsanity.com/paper/2003.10286/) (2020)

## Built on by (most cited first, 30 of 35)

- [A Comprehensive Survey of Foundation Models in Medicine](https://mlsanity.com/paper/2406.10729/) (2024)
- [KG-Rank: Enhancing Large Language Models for Medical QA with Knowledge Graphs and Ranking Techniques](https://mlsanity.com/paper/2403.05881/) (2024)
- [A Unified Review of Deep Learning for Automated Medical Coding](https://mlsanity.com/paper/2201.02797/) (2022)
- [Enhancing Supply Chain Visibility with Knowledge Graphs and Large Language Models](https://mlsanity.com/paper/2408.07705/) (2024)
- [Explainability in Practice: A Survey of Explainable NLP Across Various Domains](https://mlsanity.com/paper/2502.00837/) (2025)
- [Integrating UMLS Knowledge into Large Language Models for Medical Question Answering](https://mlsanity.com/paper/2310.02778/) (2023)
- [A Review on Generative AI Models for Synthetic Medical Text, Time Series, and Longitudinal Data](https://mlsanity.com/paper/2411.12274/) (2024)
- [Clinical NLP with Attention-Based Deep Learning for Multi-Disease Prediction](https://mlsanity.com/paper/2507.01437/) (2025)
- [PULSAR: Pre-training with Extracted Healthcare Terms for Summarising Patients' Problems and Data Augmentation with Black-box Large Language Models](https://mlsanity.com/paper/2306.02754/) (2023)
- [A Large Language Model and Denoising Diffusion Framework for Targeted Design of Microstructures with Commands in Natural Language](https://mlsanity.com/paper/2409.14473/) (2024)
- [An evaluation of DeepSeek Models in Biomedical Natural Language Processing](https://mlsanity.com/paper/2503.00624/) (2025)
- [Linking Surface Facts to Large-Scale Knowledge Graphs](https://mlsanity.com/paper/2310.14909/) (2023)
- [The Case for Globalizing Fairness: A Mixed Methods Study on Colonialism, AI, and Health in Africa](https://mlsanity.com/paper/2403.03357/) (2024)
- [Conceptualizing Machine Learning for Dynamic Information Retrieval of Electronic Health Record Notes](https://mlsanity.com/paper/2308.08494/) (2023)
- [Ascle: A Python Natural Language Processing Toolkit for Medical Text Generation](https://mlsanity.com/paper/2311.16588/) (2023)
- [Predicting Failures of LLMs to Link Biomedical Ontology Terms to Identifiers Evidence Across Models and Ontologies](https://mlsanity.com/paper/2509.04458/) (2025)
- [Patient Outcome and Zero-shot Diagnosis Prediction with Hypernetwork-guided Multitask Learning](https://mlsanity.com/paper/2109.03062/) (2021)
- [Fact or Guesswork? Evaluating Large Language Models' Medical Knowledge with Structured One-Hop Judgments](https://mlsanity.com/paper/2502.14275/) (2025)
- [Retrieval-Augmented Framework for LLM-Based Clinical Decision Support](https://mlsanity.com/paper/2510.01363/) (2025)
- [EHRKit: A Python Natural Language Processing Toolkit for Electronic Health Record Texts](https://mlsanity.com/paper/2204.06604/) (2022)
- [Towards Neural Numeric-To-Text Generation From Temporal Personal Health Data](https://mlsanity.com/paper/2207.05194/) (2022)
- [UMLS-KGI-BERT: Data-Centric Knowledge Integration in Transformers for Biomedical Entity Recognition](https://mlsanity.com/paper/2307.11170/) (2023)
- [Coupling Symbolic Reasoning with Language Modeling for Efficient Longitudinal Understanding of Unstructured Electronic Medical Records](https://mlsanity.com/paper/2308.03360/) (2023)
- [A Federated Random Forest Solution for Secure Distributed Machine Learning](https://mlsanity.com/paper/2505.08085/) (2025)
- [OpenMed NER: Open-Source, Domain-Adapted State-of-the-Art Transformers for Biomedical NER Across 12 Public Datasets](https://mlsanity.com/paper/2508.01630/) (2025)
- [Globalizing Fairness Attributes in Machine Learning: A Case Study on Health in Africa](https://mlsanity.com/paper/2304.02190/) (2023)
- [Can-SAVE: Deploying Low-Cost and Population-Scale Cancer Screening via Survival Analysis Variables and EHR](https://mlsanity.com/paper/2309.15039/) (2023)
- [Cross-Institutional Dental EHR Entity Extraction via Generative AI and Synthetic Notes](https://mlsanity.com/paper/2407.21050/) (2024)
- [Preserving Empirical Probabilities in BERT for Small-sample Clinical Entity Recognition](https://mlsanity.com/paper/2409.03238/) (2024)
- [Trust and Dependability in Blockchain & AI Based MedIoT Applications: Research Challenges and Future Directions](https://mlsanity.com/paper/2501.02647/) (2025)

## Abstract (arXiv)

Electronic health records (EHRs), digital collections of patient healthcare events and observations, are ubiquitous in medicine and critical to healthcare delivery, operations, and research. Despite this central role, EHRs are notoriously difficult to process automatically. Well over half of the information stored within EHRs is in the form of unstructured text (e.g. provider notes, operation reports) and remains largely untapped for secondary use. Recently, however, newer neural network and deep learning approaches to Natural Language Processing (NLP) have made considerable advances, outperforming traditional statistical and rule-based systems on a variety of tasks. In this survey paper, we summarize current neural NLP methods for EHR applications. We focus on a broad scope of tasks, namely, classification and prediction, word embeddings, extraction, generation, and other topics such as question answering, phenotyping, knowledge graphs, medical dialogue, multilinguality, interpretability, etc.