# Quantification of BERT Diagnosis Generalizability Across Medical Specialties Using Semantic Dataset Distance

arXiv: https://arxiv.org/abs/2008.06606 · HTML: https://mlsanity.com/paper/2008.06606/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Aug 2020.

## Built on by (most cited first, 6 of 6)

- [Structured prompt interrogation and recursive extraction of semantics (SPIRES): A method for populating knowledge bases using zero-shot learning](https://mlsanity.com/paper/2304.02711/) (2023)
- [A Cross-institutional Evaluation on Breast Cancer Phenotyping NLP Algorithms on Electronic Health Records](https://mlsanity.com/paper/2303.08448/) (2023)
- [Entity Recognition from Colloquial Text](https://mlsanity.com/paper/2401.04853/) (2024)
- [The NLP Sandbox: an efficient model-to-data system to enable federated and unbiased evaluation of clinical NLP models](https://mlsanity.com/paper/2206.14181/) (2022)
- [An Interpretable Joint Nonnegative Matrix Factorization-Based Point Cloud Distance Measure](https://mlsanity.com/paper/2207.05112/) (2022)
- [Application of Transformers based methods in Electronic Medical Records: A Systematic Literature Review](https://mlsanity.com/paper/2304.02768/) (2023)

## Abstract (arXiv)

Deep learning models in healthcare may fail to generalize on data from unseen corpora. Additionally, no quantitative metric exists to tell how existing models will perform on new data. Previous studies demonstrated that NLP models of medical notes generalize variably between institutions, but ignored other levels of healthcare organization. We measured SciBERT diagnosis sentiment classifier generalizability between medical specialties using EHR sentences from MIMIC-III. Models trained on one specialty performed better on internal test sets than mixed or external test sets (mean AUCs 0.92, 0.87, and 0.83, respectively; p = 0.016). When models are trained on more specialties, they have better test performances (p < 1e-4). Model performance on new corpora is directly correlated to the similarity between train and test sentence content (p < 1e-4). Future studies should assess additional axes of generalization to ensure deep learning models fulfil their intended purpose across institutions, specialties, and practices.