mlsanityData to 7 Oct 2026

BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text

arXiv 2403.18421 ↗First posted 27 Mar 2024cs.CL cs.AIcited by 85

Not on the map. Why?

Abstract (arXiv)

Models such as GPT-4 and Med-PaLM 2 have demonstrated impressive performance on a wide variety of biomedical NLP tasks. However, these models have hundreds of billions of parameters, are computationally expensive to run, require users to send their input data over the internet, and are trained on unknown data sources. Can smaller, more targeted models compete? To address this question, we build and release BioMedLM, a 2.7 billion parameter GPT-style autoregressive model trained exclusively on PubMed abstracts and full articles. When fine-tuned, BioMedLM can produce strong multiple-choice biomedical question-answering results competitive with much larger models, such as achieving a score of 57.3% on MedMCQA (dev) and 69.0% on the MMLU Medical Genetics exam. BioMedLM can also be fine-tuned to produce useful answers to patient questions on medical topics. This demonstrates that smaller models can potentially serve as transparent, privacy-preserving, economical and environmentally friendly foundations for particular NLP applications, such as in biomedicine. The model is available on the Hugging Face Hub: https://huggingface.co/stanford-crfm/BioMedLM.

Key facts. Cited by 85 papers in the map (data through 7 Oct 2026). Builds on 37 of its 66 references that are in the map. First posted on arXiv 27 Mar 2024.

Builds on

37 of 66 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  10. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  11. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  12. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
25 more
  1. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  2. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  3. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  4. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  5. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  6. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  7. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  8. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  9. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  10. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  11. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  12. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 319
  13. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
  14. 2022LinkBERT: Pretraining Language Models with Document Linkscited by 209
  15. 2022Compute Trends Across Three Eras of Machine Learningcited by 161
  16. 2022Deep Bidirectional Language-Knowledge Graph Pretrainingcited by 156
  17. 2020The Cost of Training NLP Models: A Concise Overviewcited by 97
  18. 2023Open-Ended Medical Visual Question Answering Through Prefix Tuning of Language Modelscited by 58
  19. 2018Learning to Summarize Radiology Findingscited by 54
  20. 2023Understanding accountability in algorithmic supply chainscited by 39
  21. 2021Efficient Domain Adaptation of Language Models via Adaptive Tokenizationcited by 34
  22. 2023Appraising the Potential Uses and Harms of LLMs for Medical Systematic Reviewscited by 18
  23. 2021MedKnowts: Unified Documentation and Information Retrieval for Electronic Health Recordscited by 7
  24. 2023Informed Named Entity Recognition Decoding for Generative Language Modelscited by 6
  25. 2023Comparative Performance Evaluation of Large Language Models for Extracting Molecular Interactions and Pathway Knowledgecited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 36; 2025: 31; 2026: 18.

85 papers.

Sort:

By area:

  1. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 96
  2. 2025From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discoverycited by 74
  3. 2024Improving Retrieval-Augmented Generation in Medicine with Iterative Follow-up Questionscited by 54
  4. 2025Injecting Domain-Specific Knowledge into Large Language Models: A Comprehensive Surveycited by 47
  5. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  6. 2025Self-Evolving Multi-Agent Simulations for Realistic Clinical Interactionscited by 40
  7. 2024SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domaincited by 39
  8. 2024SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literaturecited by 37
  9. 2024MedAdapter: Efficient Test-Time Adaptation of Large Language Models towards Medical Reasoningcited by 35
  10. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 34
  11. 2024Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress?cited by 28
  12. 2024Large Language Models for Manufacturingcited by 24
38 more
  1. 2024From Medprompt to o1: Exploration of Run-Time Strategies for Medical Challenge Problems and Beyondcited by 24
  2. 2025OmniScience: A Domain-Specialized LLM for Scientific Reasoning and Discoverycited by 20
  3. 2025Multimodal Large Language Models for Medicine: A Comprehensive Surveycited by 16
  4. 2024Towards Reliable Medical Question Answering: Techniques and Challenges in Mitigating Hallucinations in Language Modelscited by 15
  5. 2024Do We Need Domain-Specific Embedding Models? An Empirical Investigationcited by 12
  6. 2025MedCite: Can Language Models Generate Verifiable Text for Medicine?cited by 11
  7. 2025CXR-LT 2024: A MICCAI challenge on long-tailed, multi-label, and zero-shot disease classification from chest X-raycited by 10
  8. 2025Dynamic Uncertainty-aware Multimodal Fusion for Outdoor Health Monitoringcited by 10
  9. 2024LLMs are not Zero-Shot Reasoners for Biomedical Information Extractioncited by 9
  10. 2024Multi-View and Multi-Scale Alignment for Contrastive Language-Image Pre-training in Mammographycited by 9
  11. 2024The Limited Impact of Medical Adaptation of Large Language and Vision-Language Modelscited by 9
  12. 2025MedOrch: Medical Diagnosis with Tool-Augmented Reasoning Agents for Flexible Extensibilitycited by 9
  13. 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Reviewcited by 9
  14. 2024Geneverse: A collection of Open-source Multimodal Large Language Models for Genomic and Proteomic Researchcited by 8
  15. 2024Towards Building Specialized Generalist AI with System 1 and System 2 Fusioncited by 8
  16. 2024BioMamba: Domain-Adaptive Biomedical Language Modelscited by 8
  17. 2024KidLM: Advancing Language Models for Children -- Early Insights and Future Directionscited by 7
  18. 2024From Intention To Implementation: Automating Biomedical Research via LLMscited by 7
  19. 2024MedMax: Mixed-Modal Instruction Tuning for Training Biomedical Assistantscited by 6
  20. 2025Constructing Ophthalmic MLLM for Positioning-diagnosis Collaboration Through Clinical Cognitive Chain Reasoningcited by 6
  21. 2024Domain-Specific Pretraining of Language Models: A Comparative Study in the Medical Fieldcited by 5
  22. 2025Instruction Tuning and CoT Prompting for Contextual Medical QA with LLMscited by 5
  23. 2024Assessing The Potential Of Mid-Sized Language Models For Clinical QAcited by 4
  24. 2025A Short Survey on Small Reasoning Models: Training, Inference, Applications and Research Directionscited by 4
  25. 2026scPilot: Large Language Model Reasoning Toward Automated Single-Cell Analysis and Discoverycited by 4
  26. 2024AlzheimerRAG: Multimodal Retrieval Augmented Generation for Clinical Use Cases using PubMed articlescited by 3
  27. 2025PMOA-TTS: Introducing the PubMed Open Access Textual Times Series Corpuscited by 3
  28. 2025Towards EnergyGPT: A Large Language Model Specialized for the Energy Sectorcited by 3
  29. 2025BiCA: Effective Biomedical Dense Retrieval with Citation-Aware Hard Negativescited by 3
  30. 2024OnlySportsLM: Optimizing Sports-Domain Language Models with SOTA Performance under Billion Parameterscited by 2
  31. 2024The Potential of LLMs in Medical Education: Generating Questions and Answers for Qualification Examscited by 2
  32. 2024LaB-RAG: Label Boosted Retrieval Augmented Generation for Radiology Report Generationcited by 2
  33. 2024MedG-KRP: Medical Graph Knowledge Representation Probingcited by 2
  34. 2025Multimodal Cancer Modeling in the Age of Foundation Model Embeddingscited by 2
  35. 2025When Content is Goliath and Algorithm is David: The Style and Semantic Effects of Generative Search Enginecited by 2
  36. 2026Automatic Prompt Optimization for Dataset-Level Feature Discoverycited by 2
  37. 2026From Token to Token Pair: Efficient Prompt Compression for Large Language Models in Clinical Predictioncited by 2
  38. 2024A Survey for Large Language Models in Biomedicinecited by 1

50 most cited of 85 All 85 (JSON)