mlsanityData to 14 Oct 2026

BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model

arXiv 2204.03905 ↗First posted 8 Apr 2022cs.CLcited by 83

Not on the map. Why?

Abstract (arXiv)

Pretrained language models have served as important backbones for natural language processing. Recently, in-domain pretraining has been shown to benefit various domain-specific downstream tasks. In the biomedical domain, natural language generation (NLG) tasks are of critical importance, while understudied. Approaching natural language understanding (NLU) tasks as NLG achieves satisfying performance in the general domain through constrained language generation or language prompting. We emphasize the lack of in-domain generative language models and the unsystematic generative downstream benchmarks in the biomedical domain, hindering the development of the research community. In this work, we introduce the generative language model BioBART that adapts BART to the biomedical domain. We collate various biomedical language generation tasks including dialogue, summarization, entity linking, and named entity recognition. BioBART pretrained on PubMed abstracts has enhanced performance compared to BART and set strong baselines on several tasks. Furthermore, we conduct ablation studies on the pretraining tasks for BioBART and find that sentence permutation has negative effects on downstream tasks.

Key facts. Cited by 83 papers in the map (data through 14 Oct 2026). Builds on 44 of its 80 references that are in the map. First posted on arXiv 8 Apr 2022.

Builds on

44 of 80 references.

References · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  9. 2018Deep contextualized word representationscited by 2,290
  10. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  11. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  12. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
32 more
  1. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  2. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  3. 2019Publicly Available Clinical BERT Embeddingscited by 1,002
  4. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  5. 2019Text Summarization with Pretrained Encoderscited by 556
  6. 2016Neural Architectures for Named Entity Recognitioncited by 550
  7. 2020Autoregressive Entity Retrievalcited by 334
  8. 2020Self-Alignment Pretraining for Biomedical Entity Representationscited by 237
  9. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 236
  10. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  11. 2021A Unified Generative Framework for Various NER Subtaskscited by 141
  12. 2020GSum: A General Framework for Guided Neural Abstractive Summarizationcited by 129
  13. 2018Construction of the Literature Graph in Semantic Scholarcited by 118
  14. 2021CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmarkcited by 101
  15. 2020Entities as Experts: Sparse Memory Access with Entity Supervisioncited by 100
  16. 2021SciFive: a text-to-text transformer model for biomedical literaturecited by 99
  17. 2021Unified Named Entity Recognition as Word-Word Relation Classificationcited by 90
  18. 2019MedMentions: A Large Biomedical Corpus Annotated with UMLS Conceptscited by 80
  19. 2020BioMegatron: Larger Biomedical Domain Language Modelcited by 76
  20. 2020Biomedical Entity Representations with Synonym Marginalizationcited by 61
  21. 2020CODER: Knowledge infused cross-lingual medical term embedding for term normalizationcited by 55
  22. 2020COMETA: A Corpus for Medical Entity Linking in the Social Mediacited by 50
  23. 2021Biomedical Question Answering: A Survey of Approaches and Challengescited by 46
  24. 2019Probing Biomedical Embeddings from Language Modelscited by 39
  25. 2020Question-Driven Summarization of Answers to Consumer Health Questionscited by 39
  26. 2021Paradigm Shift in Natural Language Processingcited by 39
  27. 2021Improving Biomedical Pretrained Language Models with Knowledgecited by 36
  28. 2020An Empirical Study of Multi-Task Learning on BERT for Biomedical Text Miningcited by 35
  29. 2019Sequence-to-Nuggets: Nested Entity Mention Detection via Anchor-Region Networkscited by 33
  30. 2021Domain Adaptation with Pre-trained Transformers for Query Focused Abstractive Text Summarizationcited by 21
  31. 2021Cross-Domain Data Integration for Named Entity Disambiguation in Biomedical Textcited by 15
  32. 2021BERT might be Overkill: A Tiny but Effective Biomedical Entity Linker based on Residual Convolutional Neural Networkscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 31; 2024: 25; 2025: 14; 2026: 7.

83 papers.

Sort:

By area:

  1. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 301
  2. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Taskscited by 201
  3. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 163
  4. 2023Large language models in healthcare and medical domain: A reviewcited by 127
  5. 2023Benchmarking large language models for biomedical natural language processing applications and recommendationscited by 78
  6. 2022Fengshenbang 1.0: Being the Foundation of Chinese Cognitive Intelligencecited by 70
  7. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 45
  8. 2023A Comprehensive Evaluation of Large Language Models on Benchmark Biomedical Text Processing Taskscited by 36
  9. 2024A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directionscited by 34
  10. 2024Toward Foundation Model for Multivariate Wearable Sensing of Physiological Signalscited by 32
  11. 2023RAMM: Retrieval-augmented Biomedical Visual Question Answering with Multi-modal Pre-trainingcited by 31
  12. 2023nach0: Multimodal Natural and Chemical Languages Foundation Modelcited by 26
38 more
  1. 2023Evaluation of ChatGPT on Biomedical Tasks: A Zero-Shot Comparison with Fine-Tuned Generative Transformerscited by 22
  2. 2023MedEval: A Multi-Level, Multi-Task, and Multi-Domain Medical Benchmark for Language Model Evaluationcited by 19
  3. 2024Unveiling LLM Evaluation Focused on Metrics: Challenges and Solutionscited by 18
  4. 2024Overview of the BioLaySumm 2024 Shared Task on the Lay Summarization of Biomedical Research Articlescited by 18
  5. 2025Protein Large Language Models: A Comprehensive Surveycited by 18
  6. 2024Leveraging Biomolecule and Natural Language through Multi-Modal Learning: A Surveycited by 17
  7. 2024A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industrycited by 17
  8. 2023DocLens: Multi-aspect Fine-grained Evaluation for Medical Text Generationcited by 16
  9. 2025BioProBench: A Corpus and Benchmark for Biological Protocol Reasoning in Autonomous Sciencecited by 16
  10. 2022Toward expanding the scope of radiology report summarization to multiple anatomies and modalitiescited by 14
  11. 2023NapSS: Paragraph-level Medical Text Simplification via Narrative Prompting and Sentence-matching Summarizationcited by 13
  12. 2024Progress and Opportunities of Foundation Models in Bioinformaticscited by 13
  13. 2023Hierarchical Pretraining on Multimodal Electronic Health Recordscited by 11
  14. 2023Improving Biomedical Abstractive Summarisation with Knowledge Aggregation from Citation Paperscited by 9
  15. 2023FaMeSumm: Investigating and Improving Faithfulness of Medical Summarizationcited by 9
  16. 2023Less Likely Brainstorming: Using Language Models to Generate Alternative Hypothesescited by 8
  17. 2023Investigating Large Language Models and Control Mechanisms to Improve Text Readability of Biomedical Abstractscited by 8
  18. 2024MEDVOC: Vocabulary Adaptation for Fine-tuning Pre-trained Language Models on Medical Text Summarizationcited by 8
  19. 2024BioMamba: Domain-Adaptive Biomedical Language Modelscited by 8
  20. 2024No perspective, no perception!! Perspective-aware Healthcare Answer Summarizationcited by 7
  21. 2025Toward Scientific Reasoning in LLMs: Training from Expert Discussions via Reinforcement Learningcited by 7
  22. 2022A Large-Scale Dataset for Biomedical Keyphrase Generationcited by 6
  23. 2023Overview of the BioLaySumm 2023 Shared Task on Lay Summarization of Biomedical Research Articlescited by 6
  24. 2023Collaborative Synthesis of Patient Records through Multi-Visit Health State Inferencecited by 6
  25. 2025Performance Evaluation of Large Language Models in Bangla Consumer Health Query Summarizationcited by 6
  26. 2025Scaling Up Biomedical Vision-Language Models: Fine-Tuning, Instruction Tuning, and Multi-Modal Learningcited by 6
  27. 2023BioAug: Conditional Generation based Data Augmentation for Low-Resource Biomedical NERcited by 5
  28. 2023RadLing: Towards Efficient Radiology Report Understandingcited by 5
  29. 2022Enriching Biomedical Knowledge for Low-resource Language Through Large-Scale Translationcited by 4
  30. 2024Comparative Analysis of Open-Source Language Models in Summarizing Medical Text Datacited by 4
  31. 2025From Knowledge Generation to Knowledge Verification: Examining the BioMedical Generative Capabilities of ChatGPTcited by 4
  32. 2025Evaluation of LLMs in Medical Text Summarization: The Role of Vocabulary Adaptation in High OOV Settingscited by 4
  33. 2026EPPCMinerBen: A Novel Benchmark for Evaluating Large Language Models on Electronic Patient-Provider Communication via the Patient Portalcited by 4
  34. 2022DORE: Document Ordered Relation Extraction based on Generative Frameworkcited by 3
  35. 2023A Survey for Biomedical Text Summarization: From Pre-trained to Large Language Modelscited by 3
  36. 2023Ascle: A Python Natural Language Processing Toolkit for Medical Text Generationcited by 3
  37. 2024ClinLinker: Medical Entity Linking of Clinical Concept Mentions in Spanishcited by 3
  38. 2025ConTextual: Improving Clinical Text Summarization in LLMs with Context-preserving Token Filtering and Knowledge Graphscited by 3

50 most cited of 83 All 83 (JSON)