mlsanityData to 14 Oct 2026

Domain-Specific Pretraining of Language Models: A Comparative Study in the Medical Field

arXiv 2407.14076 ↗First posted 19 Jul 2024cs.LG cs.AI cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

There are many cases where LLMs are used for specific tasks in a single domain. These usually require less general, but more domain-specific knowledge. Highly capable, general-purpose state-of-the-art language models like GPT-4 or Claude-3-opus can often be used for such tasks, but they are very large and cannot be run locally, even if they were not proprietary. This can be a problem when working with sensitive data. This paper focuses on domain-specific and mixed-domain pretraining as potentially more efficient methods than general pretraining for specialized language models. We will take a look at work related to domain-specific pretraining, specifically in the medical area, and compare benchmark results of specialized language models to general-purpose language models.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 16 of its 22 references that are in the map. First posted on arXiv 19 Jul 2024.

Builds on

16 of 22 references.

References · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  5. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  6. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  7. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  8. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  9. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 638
  10. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 543
  11. 2022The Stack: 3 TB of permissively licensed source codecited by 359
  12. 2020MedDialog: Two Large-scale Medical Dialogue Datasetscited by 161
4 more
  1. 2024BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Textcited by 86
  2. 2024Apollo: A Lightweight Multilingual Medical LLM towards Democratizing Medical AI to 6B Peoplecited by 38
  3. 2024A Continued Pretrained LLM Approach for Automatic Medical Note Generationcited by 30
  4. 2020MeDAL: Medical Abbreviation Disambiguation Dataset for Natural Language Understanding Pretrainingcited by 18

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 4.

5 papers.

  1. 2024Task-Adaptive Pretrained Language Models via Clustered-Importance Samplingcited by 18
  2. 2025Optimizing Retrieval Strategies for Financial Question Answering Documents in Retrieval-Augmented Generation Systemscited by 9
  3. 2025Benchmarking Chinese Medical LLMs: A Medbench-based Analysis of Performance Gaps and Hierarchical Optimization Strategiescited by 2
  4. 2025From Amateur to Master: Infusing Knowledge into LLMs via Automated Curriculum Learningcited by 2
  5. 2025NeuroLit Navigator: A Neurosymbolic Approach to Scholarly Article Searches for Systematic Reviewscited by 0