mlsanityData to 14 Oct 2026

Probing Biomedical Embeddings from Language Models

arXiv 1904.02181 ↗First posted 3 Apr 2019cs.CLcited by 39

Not on the map. Why?

Abstract (arXiv)

Contextualized word embeddings derived from pre-trained language models (LMs) show significant improvements on downstream NLP tasks. Pre-training on domain-specific corpora, such as biomedical articles, further improves their performance. In this paper, we conduct probing experiments to determine what additional information is carried intrinsically by the in-domain trained contextualized embeddings. For this we use the pre-trained LMs as fixed feature extractors and restrict the downstream task models to not have additional sequence modeling layers. We compare BERT, ELMo, BioBERT and BioELMo, a biomedical version of ELMo trained on 10M PubMed abstracts. Surprisingly, while fine-tuned BioBERT is better than BioELMo in biomedical NER and NLI tasks, as a fixed feature extractor BioELMo outperforms BioBERT in our probing tasks. We use visualization and nearest neighbor analysis to show that better encoding of entity-type and relational information leads to this superiority.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). First posted on arXiv 3 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 12; 2022: 13; 2023: 7; 2024: 5; 2025: 2.

39 papers.

Sort:

By area:

  1. 2023Large language models in healthcare and medical domain: A reviewcited by 127
  2. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  3. 2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87
  4. 2022BioBART: Pretraining and Evaluation of A Biomedical Generative Language Modelcited by 83
  5. 2023ProtST: Multi-Modality Learning of Protein Sequences and Biomedical Textscited by 78
  6. 2021Biomedical Question Answering: A Survey of Approaches and Challengescited by 46
  7. 2021Improving Biomedical Pretrained Language Models with Knowledgecited by 36
  8. 2023MedEval: A Multi-Level, Multi-Task, and Multi-Domain Medical Benchmark for Language Model Evaluationcited by 19
  9. 2024Unveiling LLM Evaluation Focused on Metrics: Challenges and Solutionscited by 18
  10. 2025How Far Are AI Scientists from Changing the World?cited by 15
  11. 2021Benchmarking for Biomedical Natural Language Processing Tasks with a Domain Specific ALBERTcited by 14
  12. 2021Natural Language Processing for Smart Healthcarecited by 14
27 more
  1. 2023Hierarchical Pretraining on Multimodal Electronic Health Recordscited by 11
  2. 2021Probing Pre-Trained Language Models for Disease Knowledgecited by 10
  3. 2024A New Era in Computational Pathology: A Survey on Foundation and Vision-Language Modelscited by 9
  4. 2022SocioProbe: What, When, and Where Language Models Learn about Sociodemographicscited by 7
  5. 2021A Hybrid Approach to Measure Semantic Relatedness in Biomedical Conceptscited by 5
  6. 2022Transformers and the representation of biomedical background knowledgecited by 5
  7. 2021Stress Test Evaluation of Biomedical Word Embeddingscited by 4
  8. 2022Deep Learning, Natural Language Processing, and Explainable Artificial Intelligence in the Biomedical Domaincited by 4
  9. 2022Do Transformers Encode a Foundational Ontology? Probing Abstract Classes in Natural Languagecited by 3
  10. 2022Using Machine Learning to Fuse Verbal Autopsy Narratives and Binary Features in the Analysis of Deaths from Hyperglycaemiacited by 3
  11. 2023Embroid: Unsupervised Prediction Smoothing Can Improve Few-Shot Classificationcited by 3
  12. 2021Penn-Helsinki Parsed Corpus of Early Modern English: First Parsing Results and Analysiscited by 2
  13. 2022Assessment of contextualised representations in detecting outcome phrases in clinical trialscited by 2
  14. 2022Zero-Shot and Few-Shot Classification of Biomedical Articles in Context of the COVID-19 Pandemiccited by 1
  15. 2022Sentence Ambiguity, Grammaticality and Complexity Probescited by 1
  16. 2022Improving Cause-of-Death Classification from Verbal Autopsy Reportscited by 1
  17. 2023Multi-domain improves out-of-distribution and data-limited scenarios for medical image analysiscited by 1
  18. 2024Towards Efficient Methods in Medical Question Answering using Knowledge Graph Embeddingscited by 1
  19. 2021Let Your Heart Speak in its Mother Tongue: Multilingual Captioning of Cardiac Signalscited by 0
  20. 2021Transformer-based Language Models for Factoid Question Answering at BioASQ9bcited by 0
  21. 2021Recent Advances in Automated Question Answering In Biomedical Domaincited by 0
  22. 2022ABB-BERT: A BERT model for disambiguating abbreviations and contractionscited by 0
  23. 2022Probing of Quantitative Values in Abstractive Summarization Modelscited by 0
  24. 2022Token Classification for Disambiguating Medical Abbreviationscited by 0
  25. 2023Towards Probing Contact Center Large Language Modelscited by 0
  26. 2024Named Clinical Entity Recognition Benchmarkcited by 0
  27. 2025Hierarchical Multi-Label Contrastive Learning for Protein-Protein Interaction Prediction Across Organismscited by 0