mlsanityData to 14 Oct 2026

BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains

arXiv 2402.10373 ↗First posted 15 Feb 2024cs.CL cs.AI cs.LGcited by 301

Not on the map. Why?

Cited by 34 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable versatility in recent years, offering potential applications across specialized domains such as healthcare and medicine. Despite the availability of various open-source LLMs tailored for health contexts, adapting general-purpose LLMs to the medical domain presents significant challenges. In this paper, we introduce BioMistral, an open-source LLM tailored for the biomedical domain, utilizing Mistral as its foundation model and further pre-trained on PubMed Central. We conduct a comprehensive evaluation of BioMistral on a benchmark comprising 10 established medical question-answering (QA) tasks in English. We also explore lightweight models obtained through quantization and model merging approaches. Our results demonstrate BioMistral's superior performance compared to existing open-source medical models and its competitive edge against proprietary counterparts. Finally, to address the limited availability of data beyond English and to assess the multilingual generalization of medical LLMs, we automatically translated and evaluated this benchmark into 7 other languages. This marks the first large-scale multilingual evaluation of LLMs in the medical domain. Datasets, multilingual evaluation benchmarks, scripts, and all the models obtained during our experiments are freely released.

Key facts. Cited by 301 papers in the map (data through 14 Oct 2026). Builds on 43 of its 69 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

43 of 69 references.

References · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  8. 2020Longformer: The Long-Document Transformercited by 3,413
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  11. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
31 more
  1. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  2. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  3. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  4. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  5. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  6. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  7. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  8. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  9. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  10. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 638
  11. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 538
  12. 2023The Falcon Series of Open Language Modelscited by 535
  13. 2023Zephyr: Direct Distillation of LM Alignmentcited by 506
  14. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  15. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 434
  16. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 412
  17. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
  18. 2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 325
  19. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 265
  20. 2019Model Fusion via Optimal Transportcited by 219
  21. 2022Can large language models reason about medical questions?cited by 194
  22. 2022Patching open-vocabulary models by interpolating weightscited by 190
  23. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 163
  24. 2021Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalizationcited by 144
  25. 2023Evaluating and Mitigating Discrimination in Language Model Decisionscited by 124
  26. 2023Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-world Multi-turn Dialoguecited by 119
  27. 2022Fusing finetuned models for better pretrainingcited by 110
  28. 2023ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluationcited by 88
  29. 2022BioBART: Pretraining and Evaluation of A Biomedical Generative Language Modelcited by 83
  30. 2018Multi-Source Domain Adaptation with Mixture of Expertscited by 62
  31. 2023DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domainscited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 111; 2025: 102; 2026: 88.

301 papers.

Sort:

By area:

  1. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 265
  2. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 194
  3. 2025MedReason: Eliciting Factual Medical Reasoning Steps in LLMs via Knowledge Graphscited by 110
  4. 2024On-Device Language Models: A Comprehensive Reviewcited by 104
  5. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 99
  6. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  7. 2024Towards Building Multilingual Language Model for Medicinecited by 92
  8. 2024Advancing Multimodal Medical Capabilities of Geminicited by 90
  9. 2024Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Modelscited by 72
  10. 2024UltraMedical: Building Specialized Generalists in Biomedicinecited by 67
  11. 2024TensorOpera Router: A Multi-Model Router for Efficient LLM Inferencecited by 60
  12. 2024SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)cited by 59
38 more
  1. 2024Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimizationcited by 56
  2. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56
  3. 2024DALK: Dynamic Co-Augmentation of LLMs and KG to answer Alzheimer's Disease Questions with Scientific Literaturecited by 50
  4. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  5. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 45
  6. 2024MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answeringcited by 43
  7. 2024BioRAG: A RAG-LLM Framework for Biological Question Reasoningcited by 43
  8. 2024SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literaturecited by 41
  9. 2024Rationale-Guided Retrieval Augmented Generation for Medical Question Answeringcited by 41
  10. 2024TelecomGPT: A Framework to Build Telecom-Specfic Large Language Modelscited by 40
  11. 2024Towards Evaluating and Building Versatile Large Language Models for Medicinecited by 40
  12. 2024Apollo: A Lightweight Multilingual Medical LLM towards Democratizing Medical AI to 6B Peoplecited by 38
  13. 2024BMRetriever: Tuning Large Language Models as Better Biomedical Text Retrieverscited by 38
  14. 2024MedAdapter: Efficient Test-Time Adaptation of Large Language Models towards Medical Reasoningcited by 36
  15. 2025Beyond Distillation: Pushing the Limits of Medical LLM Reasoning with Minimalist Rule-Based RLcited by 36
  16. 2025MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Modelscited by 34
  17. 2025Scientists' First Exam: Probing Cognitive Abilities of MLLM via Perception, Understanding, and Reasoningcited by 34
  18. 2024JMLR: Joint Medical LLM and Retrieval Training for Enhancing Reasoning and Professional Question Answering Capabilitycited by 32
  19. 2024Small Language Models Learn Enhanced Reasoning Skills from Medical Textbookscited by 30
  20. 2024SimRAG: Self-Improving Retrieval-Augmented Generation for Adapting Large Language Models to Specialized Domainscited by 28
  21. 2024Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress?cited by 28
  22. 2025Real-World Doctor Agent with Proactive Consultation through Multi-Agent Reinforcement Learningcited by 27
  23. 2024Efficient Continual Pre-training by Mitigating the Stability Gapcited by 25
  24. 2024Tele-LLMs: A Series of Specialized Large Language Models for Telecommunicationscited by 25
  25. 2024A Benchmark for Long-Form Medical Question Answeringcited by 25
  26. 2025ALFA: Aligning LLMs to Ask Good Questions A Case Study in Clinical Reasoningcited by 25
  27. 2024AAAR-1.0: Assessing AI's Potential to Assist Researchcited by 24
  28. 2025SFT Doesn't Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMscited by 24
  29. 2025Primus: A Pioneering Collection of Open-Source Datasets for Cybersecurity LLM Trainingcited by 23
  30. 2024LLMs4OM: Matching Ontologies with Large Language Modelscited by 22
  31. 2024A Survey of Conversational Searchcited by 22
  32. 2024Biomedical Large Languages Models Seem not to be Superior to Generalist Models on Unseen Medical Datacited by 21
  33. 2024MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimizationcited by 21
  34. 2024Aloe: A Family of Fine-tuned Open Healthcare LLMscited by 20
  35. 2024OLAPH: Improving Factuality in Biomedical Long-form Question Answeringcited by 20
  36. 2025A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignmentcited by 20
  37. 2025The Rise of Small Language Models in Healthcare: A Comprehensive Surveycited by 18
  38. 2024Leveraging Biomolecule and Natural Language through Multi-Modal Learning: A Surveycited by 17

50 most cited of 301 All 301 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey for Large Language Models in Biomedicine
  2. 2026MedLLM: An Open Medical Language Model at the Sub-Billion Scale
  3. 2023Benchmarking large language models for biomedical natural language processing applications and recommendations
  4. 2024LLM-MedQA: Enhancing Medical Question Answering through Case Studies in Large Language Models
  5. 2025MedBioLM: Optimizing Medical and Biological QA with Fine-Tuned Large Language Models and Retrieval-Augmented Generation
  6. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Models
  7. 2025PersianMedQA: Evaluating Large Language Models on a Persian-English Bilingual Medical Question Answering Benchmark
  8. 2023Evaluation of ChatGPT Family of Models for Biomedical Reasoning and Classification