mlsanityData to 14 Oct 2026

Benchmarking large language models for biomedical natural language processing applications and recommendations

arXiv 2305.16326 ↗First posted 10 May 2023cs.CL cs.AI cs.IRcited by 78

Not on the map. Why?

Abstract (arXiv)

The rapid growth of biomedical literature poses challenges for manual knowledge curation and synthesis. Biomedical Natural Language Processing (BioNLP) automates the process. While Large Language Models (LLMs) have shown promise in general domains, their effectiveness in BioNLP tasks remains unclear due to limited benchmarks and practical guidelines. We perform a systematic evaluation of four LLMs, GPT and LLaMA representatives on 12 BioNLP benchmarks across six applications. We compare their zero-shot, few-shot, and fine-tuning performance with traditional fine-tuning of BERT or BART models. We examine inconsistencies, missing information, hallucinations, and perform cost analysis. Here we show that traditional fine-tuning outperforms zero or few shot LLMs in most tasks. However, closed-source LLMs like GPT-4 excel in reasoning-related tasks such as medical question answering. Open source LLMs still require fine-tuning to close performance gaps. We find issues like missing information and hallucinations in LLM outputs. These results offer practical insights for applying LLMs in BioNLP.

Key facts. Cited by 78 papers in the map (data through 14 Oct 2026). Builds on 40 of its 103 references that are in the map. First posted on arXiv 10 May 2023.

Builds on

40 of 103 references.

References · 40

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  5. 2023A Survey of Large Language Modelscited by 2,576
  6. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  7. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  8. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  9. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  10. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  11. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  12. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 638
28 more
  1. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 538
  2. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 478
  3. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 361
  4. 2019Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasetscited by 299
  5. 2022Large Language Models are Few-Shot Clinical Information Extractorscited by 207
  6. 2022Deep Bidirectional Language-Knowledge Graph Pretrainingcited by 156
  7. 2021SciFive: a text-to-text transformer model for biomedical literaturecited by 99
  8. 2023Do We Still Need Clinical Language Models?cited by 98
  9. 2022Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Againcited by 83
  10. 2022BioBART: Pretraining and Evaluation of A Biomedical Generative Language Modelcited by 83
  11. 2021MS2: Multi-Document Summarization of Medical Studiescited by 80
  12. 2022An Empirical Survey on Long Document Summarization: Datasets, Models and Metricscited by 74
  13. 2023How Robust is GPT-3.5 to Predecessors? A Comprehensive Study on Language Understanding Taskscited by 58
  14. 2021Fine-Tuning Large Neural Language Models for Biomedical Natural Language Processingcited by 57
  15. 2021Paragraph-level Simplification of Medical Textscited by 53
  16. 2018BioSentVec: creating sentence embeddings for biomedical textscited by 47
  17. 2021Biomedical Question Answering: A Survey of Approaches and Challengescited by 46
  18. 2022Optimizing Bi-Encoder for Named Entity Recognition via Contrastive Learningcited by 24
  19. 2022Readability Controllable Biomedical Document Summarizationcited by 19
  20. 2018ML-Net: multi-label classification of biomedical texts with deep neural networkscited by 17
  21. 2022Multi-label classification for biomedical literature: an overview of the BioCreative VII LitCovid Track for COVID-19 literature topic annotationscited by 14
  22. 2022GRETEL: Graph Contrastive Topic Enhanced Language Model for Long Document Extractive Summarizationcited by 10
  23. 2023Bioformer: an efficient transformer language model for biomedical text miningcited by 9
  24. 2023Evaluation of GPT and BERT-based models on identifying protein-protein interactions in biomedical textcited by 9
  25. 2019BioConceptVec: creating and evaluating literature-based biomedical concept embeddings on a large scalecited by 8
  26. 2022LitCovid in 2022: an information resource for the COVID-19 literaturecited by 7
  27. 2022LitMC-BERT: transformer-based multi-label classification of biomedical literature with an application on COVID-19 literature curationcited by 6
  28. 2022Comprehensively identifying Long Covid articles with human-in-the-loop machine learningcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 10; 2024: 12; 2025: 29; 2026: 27.

78 papers.

Sort:

By area:

  1. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 331
  2. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 124
  3. 2023Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Healthcited by 118
  4. 2025MedicalAgentsBench for Complex Medical Reasoning: Comparing Internalized Reasoning Models versus Externalized Agent-based Frameworkscited by 40
  5. 2023Taiyi: A Bilingual Fine-Tuned Large Language Model for Diverse Biomedical Taskscited by 36
  6. 2024MedAdapter: Efficient Test-Time Adaptation of Large Language Models towards Medical Reasoningcited by 36
  7. 2023An Empirical Evaluation of Prompting Strategies for Large Language Models in Zero-Shot Clinical Natural Language Processingcited by 19
  8. 2023Identifying and Extracting Rare Disease Phenotypes with Large Language Modelscited by 18
  9. 2024Unveiling LLM Evaluation Focused on Metrics: Challenges and Solutionscited by 18
  10. 2025MultiFinBen: Benchmarking Large Language Models for Multilingual and Multimodal Financial Applicationcited by 14
  11. 2025BRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice Textcited by 12
  12. 2023An evaluation of GPT models for phenotype concept recognitioncited by 10
38 more
  1. 2023Large Language Models Vote: Prompting for Rare Disease Identificationcited by 9
  2. 2024Information Extraction from Clinical Notes: Are We Ready to Switch to Large Language Models?cited by 9
  3. 2024ClinicRealm: Re-evaluating Large Language Models with Conventional Machine Learning for Non-Generative Clinical Prediction Taskscited by 8
  4. 2024BioMamba: Domain-Adaptive Biomedical Language Modelscited by 8
  5. 2026ATPO: Adaptive Tree Policy Optimization for Multi-Turn Medical Dialoguecited by 8
  6. 2025Scaling Up Biomedical Vision-Language Models: Fine-Tuning, Instruction Tuning, and Multi-Modal Learningcited by 6
  7. 2025Benchmarking GPT-5 for biomedical natural language processingcited by 5
  8. 2025DFIR-Metric: A Benchmark Dataset for Evaluating Large Language Models in Digital Forensics and Incident Responsecited by 4
  9. 2025AutoPK: Leveraging LLMs and a Hybrid Similarity Metric for Advanced Retrieval of Pharmacokinetic Data from Complex Tables and Documentscited by 4
  10. 2026EPPCMinerBen: A Novel Benchmark for Evaluating Large Language Models on Electronic Patient-Provider Communication via the Patient Portalcited by 4
  11. 2026QCAgent: An agentic framework for quality-controllable pathology report generation from whole slide imagecited by 4
  12. 2024A comparative study of zero-shot inference with large language models and supervised modeling in breast cancer pathology classificationcited by 3
  13. 2024Entry-level guide to the use of large language models for medical researchcited by 3
  14. 2025Quantized Large Language Models in Biomedical Natural Language Processing: Evaluation and Recommendationcited by 3
  15. 2026Measuring Epistemic Resilience of LLMs Under Misleading Medical Contextcited by 3
  16. 2023ExpNote: Black-box Large Language Models are Better Task Solvers with Experience Notebookcited by 2
  17. 2024Enhancing Large Language Models with Domain-specific Retrieval Augment Generation: A Case Study on Long-form Consumer Health Question Answering in Ophthalmologycited by 2
  18. 2025SciCUEval: A Comprehensive Dataset for Evaluating Scientific Context Understanding in Large Language Modelscited by 2
  19. 2025TAGS: A Test-Time Generalist-Specialist Framework with Retrieval-Augmented Reasoning and Verificationcited by 2
  20. 2025LLM-Based Agents for Competitive Landscape Mapping in Drug Asset Due Diligencecited by 2
  21. 2025Building Trustworthy AI for Materials Discovery: From Autonomous Laboratories to Z-scorescited by 2
  22. 2026Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluationcited by 2
  23. 2023An In-Depth Evaluation of Federated Learning on Biomedical Natural Language Processingcited by 1
  24. 2024Rapid Biomedical Research Classification: The Pandemic PACT Advanced Categorisation Enginecited by 1
  25. 2024Eir: Thai Medical Large Language Modelscited by 1
  26. 2024DiMB-RE: Mining the Scientific Literature for Diet-Microbiome Associationscited by 1
  27. 2025Enhancing Biomedical Relation Extraction with Directionalitycited by 1
  28. 2025Inceptive Transformers: Enhancing Contextual Representations through Multi-Scale Feature Learning Across Domains and Languagescited by 1
  29. 2025Scalable Medication Extraction and Discontinuation Identification from Electronic Health Records Using Large Language Modelscited by 1
  30. 2025PrivacyXray: Detecting Privacy Breaches in LLMs through Semantic Consistency and Probability Certaintycited by 1
  31. 2025HySim-LLM: Embedding-Weighted Fine-Tuning Bounds and Manifold Denoising for Domain-Adapted LLMscited by 1
  32. 2025Diagnosing Hallucination Risk in AI Surgical Decision-Support: A Sequential Framework for Sequential Validationcited by 1
  33. 2025Causal Graph Neural Networks for Healthcarecited by 1
  34. 2026SciHorizon-GENE: Benchmarking LLM for Life Sciences Inference from Gene Knowledge to Functional Understandingcited by 1
  35. 2026Towards Fair and Efficient De-identification: Quantifying the Efficiency and Generalizability of De-identification Approachescited by 1
  36. 2026Toward Reliable, Safe, and Secure LLMs for Scientific Applicationscited by 1
  37. 2026GenomeQA: Benchmarking General Large Language Models for Genome Sequence Understandingcited by 1
  38. 2026MADE: A Living Benchmark for Multi-Label Text Classification with Uncertainty Quantification of Medical Device Adverse Eventscited by 1

50 most cited of 78 All 78 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey for Large Language Models in Biomedicine
  2. 2024Biomedical Large Languages Models Seem not to be Superior to Generalist Models on Unseen Medical Data
  3. 2024Benchmarking Retrieval-Augmented Large Language Models in Biomedical NLP: Application, Robustness, and Self-Awareness
  4. 2023Evaluation of ChatGPT Family of Models for Biomedical Reasoning and Classification
  5. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains
  6. 2025Disentangling Reasoning and Knowledge in Medical Large Language Models
  7. 2025Benchmarking Large Language Models on Multiple Tasks in Bioinformatics NLP with Prompting
  8. 2026Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks