mlsanityData to 7 Oct 2026

Can Language Models be Biomedical Knowledge Bases?

arXiv 2109.07154 ↗First posted 15 Sep 2021cs.CLcited by 83

Not on the map. Why?

Abstract (arXiv)

Pre-trained language models (LMs) have become ubiquitous in solving various natural language processing (NLP) tasks. There has been increasing interest in what knowledge these LMs contain and how we can extract that knowledge, treating LMs as knowledge bases (KBs). While there has been much work on probing LMs in the general domain, there has been little attention to whether these powerful LMs can be used as domain-specific KBs. To this end, we create the BioLAMA benchmark, which is comprised of 49K biomedical factual knowledge triples for probing biomedical LMs. We find that biomedical LMs with recently proposed probing methods can achieve up to 18.51% Acc@5 on retrieving biomedical knowledge. Although this seems promising given the task difficulty, our detailed analyses reveal that most predictions are highly correlated with prompt templates without any subjects, hence producing similar results on each relation and hindering their capabilities to be used as domain-specific KBs. We hope that BioLAMA can serve as a challenging benchmark for biomedical factual probing.

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Builds on 11 of its 20 references that are in the map. First posted on arXiv 15 Sep 2021.

Builds on

11 of 20 references.

References · 11

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2018Deep contextualized word representationscited by 2,285
  3. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  4. 2019Language Models as Knowledge Bases?cited by 1,922
  5. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  6. 2019How Can We Know What Language Models Know?cited by 953
  7. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  8. 2020X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Modelscited by 117
  9. 2021Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Basescited by 107
  10. 2019E-BERT: Efficient-Yet-Effective Entity Embeddings for BERTcited by 96
  11. 2020Biomedical Entity Representations with Synonym Marginalizationcited by 59

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 16; 2023: 33; 2024: 22; 2025: 7; 2026: 4.

83 papers.

Sort:

By area:

  1. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  2. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 310
  3. 2022A Review on Language Models as Knowledge Basescited by 156
  4. 2023PaperQA: Retrieval-Augmented Generative Agent for Scientific Researchcited by 143
  5. 2023Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answeringcited by 119
  6. 2023CMB: A Comprehensive Medical Benchmark in Chinesecited by 113
  7. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 69
  8. 2023GeoLLM: Extracting Geospatial Knowledge from Large Language Modelscited by 62
  9. 2023Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modelingcited by 56
  10. 2023Prompt Engineering for Healthcare: Methodologies and Applicationscited by 52
  11. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  12. 2023Knowledge-Augmented Large Language Models for Personalized Contextual Query Suggestioncited by 43
38 more
  1. 2023BioPlanner: Automatic Evaluation of LLMs on Protocol Planning in Biologycited by 42
  2. 2022Knowledge Graph Embedding: A Survey from the Perspective of Representation Spacescited by 41
  3. 2022Knowledge Injected Prompt Based Fine-tuning for Multi-label Few-shot ICD Codingcited by 40
  4. 2023Summarizing, Simplifying, and Synthesizing Medical Evidence Using GPT-3 (with Varying Success)cited by 40
  5. 2022Prompting as Probing: Using Language Models for Knowledge Base Constructioncited by 32
  6. 2023Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Modelscited by 32
  7. 2024JMLR: Joint Medical LLM and Retrieval Training for Enhancing Reasoning and Professional Question Answering Capabilitycited by 32
  8. 2022Can Prompt Probe Pretrained Language Models? Understanding the Invisible Risks from a Causal Viewcited by 31
  9. 2022GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Modelscited by 30
  10. 2022CoVERT: A Corpus of Fact-checked Biomedical COVID-19 Tweetscited by 29
  11. 2023Language Model Analysis for Ontology Subsumption Inferencecited by 25
  12. 2023The Life Cycle of Knowledge in Big Language Models: A Surveycited by 23
  13. 2021Rewire-then-Probe: A Contrastive Recipe for Probing Biomedical Knowledge of Pre-trained Language Modelscited by 21
  14. 2023Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Viewscited by 21
  15. 2023PaniniQA: Enhancing Patient Education Through Interactive Question Answeringcited by 21
  16. 2023Large Language Models for Biomedical Knowledge Graph Construction: Information extraction from EMR notescited by 17
  17. 2023README: Bridging Medical Jargon and Lay Understanding for Patient Education through Data-Centric NLPcited by 17
  18. 2024Can Language Models Act as Knowledge Bases at Scale?cited by 17
  19. 2022Extracting Biomedical Factual Knowledge Using Pretrained Language Model and Electronic Health Record Contextcited by 15
  20. 2022Context Variance Evaluation of Pretrained Language Models for Prompt-based Biomedical Knowledge Probingcited by 15
  21. 2023Multimodal Machine Learning in Image-Based and Clinical Biomedicine: Survey and Prospectscited by 15
  22. 2024Adaptive Question Answering: Enhancing Language Model Proficiency for Addressing Knowledge Conflicts with Source Citationscited by 15
  23. 2023UMASS_BioNLP at MEDIQA-Chat 2023: Can LLMs generate high-quality synthetic note-oriented doctor-patient conversations?cited by 14
  24. 2022Domain-Specific Text Generation for Machine Translationcited by 12
  25. 2023Understanding Finetuning for Factual Knowledge Extraction from Language Modelscited by 12
  26. 2023TaskLAMA: Probing the Complex Task Understanding of Language Modelscited by 12
  27. 2024SYNFAC-EDIT: Synthetic Imitation Edit Feedback for Factual Alignment in Clinical Summarizationcited by 12
  28. 2024Identifying Query-Relevant Neurons in Large Language Models for Long-Form Textscited by 12
  29. 2023Knowledge Graphs for the Life Sciences: Recent Developments, Challenges and Opportunitiescited by 10
  30. 2024SelfPrompt: Autonomously Evaluating LLM Robustness via Domain-Constrained Knowledge Guidelines and Refined Adversarial Promptscited by 10
  31. 2025BioHopR: A Benchmark for Multi-Hop, Multi-Answer Reasoning in Biomedical Domaincited by 10
  32. 2023On the Evolution of Knowledge Graphs: A Survey and Perspectivecited by 9
  33. 2024A Survey on Symbolic Knowledge Distillation of Large Language Modelscited by 9
  34. 2022Position-based Prompting for Health Outcome Generationcited by 7
  35. 2023Improving Summarization with Human Editscited by 7
  36. 2024Desiderata for the Context Use of Question Answering Systemscited by 7
  37. 2024The Queen of England is not England's Queen: On the Lack of Factual Coherency in PLMscited by 7
  38. 2024ChroKnowledge: Unveiling Chronological Knowledge of Language Models in Multiple Domainscited by 7

50 most cited of 83 All 83 (JSON)