# Probing Biomedical Embeddings from Language Models

arXiv: https://arxiv.org/abs/1904.02181 · HTML: https://mlsanity.com/paper/1904.02181/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Apr 2019.

## Built on by (most cited first, 30 of 39)

- [Large language models in healthcare and medical domain: A review](https://mlsanity.com/paper/2401.06775/) (2023)
- [A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery](https://mlsanity.com/paper/2406.10833/) (2024)
- [Pre-trained Language Models in Biomedical Domain: A Systematic Survey](https://mlsanity.com/paper/2110.05006/) (2021)
- [BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model](https://mlsanity.com/paper/2204.03905/) (2022)
- [ProtST: Multi-Modality Learning of Protein Sequences and Biomedical Texts](https://mlsanity.com/paper/2301.12040/) (2023)
- [Biomedical Question Answering: A Survey of Approaches and Challenges](https://mlsanity.com/paper/2102.05281/) (2021)
- [Improving Biomedical Pretrained Language Models with Knowledge](https://mlsanity.com/paper/2104.10344/) (2021)
- [MedEval: A Multi-Level, Multi-Task, and Multi-Domain Medical Benchmark for Language Model Evaluation](https://mlsanity.com/paper/2310.14088/) (2023)
- [Unveiling LLM Evaluation Focused on Metrics: Challenges and Solutions](https://mlsanity.com/paper/2404.09135/) (2024)
- [How Far Are AI Scientists from Changing the World?](https://mlsanity.com/paper/2507.23276/) (2025)
- [Benchmarking for Biomedical Natural Language Processing Tasks with a Domain Specific ALBERT](https://mlsanity.com/paper/2107.04374/) (2021)
- [Natural Language Processing for Smart Healthcare](https://mlsanity.com/paper/2110.15803/) (2021)
- [Hierarchical Pretraining on Multimodal Electronic Health Records](https://mlsanity.com/paper/2310.07871/) (2023)
- [Probing Pre-Trained Language Models for Disease Knowledge](https://mlsanity.com/paper/2106.07285/) (2021)
- [A New Era in Computational Pathology: A Survey on Foundation and Vision-Language Models](https://mlsanity.com/paper/2408.14496/) (2024)
- [SocioProbe: What, When, and Where Language Models Learn about Sociodemographics](https://mlsanity.com/paper/2211.04281/) (2022)
- [A Hybrid Approach to Measure Semantic Relatedness in Biomedical Concepts](https://mlsanity.com/paper/2101.10196/) (2021)
- [Transformers and the representation of biomedical background knowledge](https://mlsanity.com/paper/2202.02432/) (2022)
- [Stress Test Evaluation of Biomedical Word Embeddings](https://mlsanity.com/paper/2107.11652/) (2021)
- [Deep Learning, Natural Language Processing, and Explainable Artificial Intelligence in the Biomedical Domain](https://mlsanity.com/paper/2202.12678/) (2022)
- [Do Transformers Encode a Foundational Ontology? Probing Abstract Classes in Natural Language](https://mlsanity.com/paper/2201.10262/) (2022)
- [Using Machine Learning to Fuse Verbal Autopsy Narratives and Binary Features in the Analysis of Deaths from Hyperglycaemia](https://mlsanity.com/paper/2204.12169/) (2022)
- [Embroid: Unsupervised Prediction Smoothing Can Improve Few-Shot Classification](https://mlsanity.com/paper/2307.11031/) (2023)
- [Penn-Helsinki Parsed Corpus of Early Modern English: First Parsing Results and Analysis](https://mlsanity.com/paper/2112.08532/) (2021)
- [Assessment of contextualised representations in detecting outcome phrases in clinical trials](https://mlsanity.com/paper/2203.03547/) (2022)
- [Zero-Shot and Few-Shot Classification of Biomedical Articles in Context of the COVID-19 Pandemic](https://mlsanity.com/paper/2201.03017/) (2022)
- [Sentence Ambiguity, Grammaticality and Complexity Probes](https://mlsanity.com/paper/2210.06928/) (2022)
- [Improving Cause-of-Death Classification from Verbal Autopsy Reports](https://mlsanity.com/paper/2210.17161/) (2022)
- [Multi-domain improves out-of-distribution and data-limited scenarios for medical image analysis](https://mlsanity.com/paper/2310.06737/) (2023)
- [Towards Efficient Methods in Medical Question Answering using Knowledge Graph Embeddings](https://mlsanity.com/paper/2401.07977/) (2024)

## Abstract (arXiv)

Contextualized word embeddings derived from pre-trained language models (LMs) show significant improvements on downstream NLP tasks. Pre-training on domain-specific corpora, such as biomedical articles, further improves their performance. In this paper, we conduct probing experiments to determine what additional information is carried intrinsically by the in-domain trained contextualized embeddings. For this we use the pre-trained LMs as fixed feature extractors and restrict the downstream task models to not have additional sequence modeling layers. We compare BERT, ELMo, BioBERT and BioELMo, a biomedical version of ELMo trained on 10M PubMed abstracts. Surprisingly, while fine-tuned BioBERT is better than BioELMo in biomedical NER and NLI tasks, as a fixed feature extractor BioELMo outperforms BioBERT in our probing tasks. We use visualization and nearest neighbor analysis to show that better encoding of entity-type and relational information leads to this superiority.