mlsanityData to 7 Oct 2026

BioBERT: a pre-trained biomedical language representation model for biomedical text mining

arXiv 1901.08746 ↗First posted 25 Jan 2019cs.CLcited by 1,935

Not on the map. Why?

Cited by 41 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Biomedical text mining is becoming increasingly important as the number of biomedical documents rapidly grows. With the progress in natural language processing (NLP), extracting valuable information from biomedical literature has gained popularity among researchers, and deep learning has boosted the development of effective biomedical text mining models. However, directly applying the advancements in NLP to biomedical text mining often yields unsatisfactory results due to a word distribution shift from general domain corpora to biomedical corpora. In this article, we investigate how the recently introduced pre-trained language model BERT can be adapted for biomedical corpora. We introduce BioBERT (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining), which is a domain-specific language representation model pre-trained on large-scale biomedical corpora. With almost the same architecture across tasks, BioBERT largely outperforms BERT and previous state-of-the-art models in a variety of biomedical text mining tasks when pre-trained on biomedical corpora. While BERT obtains performance comparable to that of previous state-of-the-art models, BioBERT significantly outperforms them on the following three representative biomedical text mining tasks: biomedical named entity recognition (0.62% F1 score improvement), biomedical relation extraction (2.80% F1 score improvement) and biomedical question answering (12.24% MRR improvement). Our analysis results show that pre-training BERT on biomedical corpora helps it to understand complex biomedical texts. We make the pre-trained weights of BioBERT freely available at https://github.com/naver/biobert-pretrained, and the source code for fine-tuning BioBERT available at https://github.com/dmis-lab/biobert.

Key facts. Cited by 1,935 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 278; 2022: 309; 2023: 402; 2024: 350; 2025: 385; 2026: 211.

1,935 papers.

Sort:

By area:

  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  2. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  3. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  4. 2023BloombergGPT: A Large Language Model for Financecited by 923
  5. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 779
  6. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  7. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  8. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  9. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  10. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  11. 2021QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answeringcited by 386
  12. 2021Efficient Attentions for Long Document Summarizationcited by 342
38 more
  1. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
  2. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 326
  3. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  4. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 311
  5. 2021RadGraph: Extracting Clinical Entities and Relations from Radiology Reportscited by 310
  6. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 299
  7. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 272
  8. 2022GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Recordscited by 238
  9. 2022LinkBERT: Pretraining Language Models with Document Linkscited by 209
  10. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 208
  11. 2022Large Language Models are Few-Shot Clinical Information Extractorscited by 206
  12. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 199
  13. 2023Continual Pre-training of Language Modelscited by 194
  14. 2023Large Language Models on Graphs: A Comprehensive Surveycited by 190
  15. 2022Cross-modal Memory Networks for Radiology Report Generationcited by 183
  16. 2023MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrievalcited by 182
  17. 2022GreaseLM: Graph REASoning Enhanced Language Models for Question Answeringcited by 170
  18. 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attackscited by 168
  19. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  20. 2021MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcarecited by 159
  21. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 157
  22. 2022Deep Bidirectional Language-Knowledge Graph Pretrainingcited by 156
  23. 2023Does Synthetic Data Generation of LLMs Help Clinical Text Mining?cited by 151
  24. 2023PaperQA: Retrieval-Augmented Generative Agent for Scientific Researchcited by 143
  25. 2021MultiVerS: Improving scientific claim verification with weak supervision and full-document contextcited by 140
  26. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  27. 2023DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Taskcited by 130
  28. 2022TimeLMs: Diachronic Language Models from Twittercited by 129
  29. 2022A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Languagecited by 129
  30. 2023DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4cited by 128
  31. 2023Large language models in healthcare and medical domain: A reviewcited by 126
  32. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  33. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  34. 2023Almanac: Retrieval-Augmented Language Models for Clinical Medicinecited by 122
  35. 2023Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Healthcited by 118
  36. 2022WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domaincited by 117
  37. 2023ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Modelscited by 116
  38. 2023ACI-BENCH: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generationcited by 116

50 most cited of 1,935 All 1,935 (JSON)