# Trade-offs in Medical LLM Adaptation: An Empirical Study in French QA

arXiv: https://arxiv.org/abs/2606.19266 · HTML: https://mlsanity.com/paper/2606.19266/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 48 references that are in the map.
- First posted on arXiv 17 Jun 2026.

## Builds on (most cited first, 27 of 27)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [JudgeLM: Fine-tuned Large Language Models are Scalable Judges](https://mlsanity.com/paper/2310.17631/) (2023)
- [ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge](https://mlsanity.com/paper/2303.14070/) (2023)
- [BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains](https://mlsanity.com/paper/2402.10373/) (2024)
- [VeRA: Vector-based Random Matrix Adaptation](https://mlsanity.com/paper/2310.11454/) (2023)
- [PMC-LLaMA: Towards Building Open-source Language Models for Medicine](https://mlsanity.com/paper/2304.14454/) (2023)
- [Fine-tuning large language models for domain adaptation: Exploration of training strategies, scaling, model merging and synergistic capabilities](https://mlsanity.com/paper/2409.03444/) (2024)
- [Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Method](https://mlsanity.com/paper/2409.14781/) (2024)
- [DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distribution](https://mlsanity.com/paper/2405.17357/) (2024)
- [Parameter-Efficient Fine-Tuning of LLaMA for the Clinical Domain](https://mlsanity.com/paper/2307.03042/) (2023)
- [K-QA: A Real-World Medical Q&A Benchmark](https://mlsanity.com/paper/2401.14493/) (2024)
- [DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domains](https://mlsanity.com/paper/2304.00958/) (2023)
- [FrenchMedMCQA: A French Multiple-Choice Question Answering Dataset for Medical domain](https://mlsanity.com/paper/2304.04280/) (2023)
- [BiMediX: Bilingual Medical Mixture of Experts LLM](https://mlsanity.com/paper/2402.13253/) (2024)
- [A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industry](https://mlsanity.com/paper/2404.15777/) (2024)
- [Medical mT5: An Open-Source Multilingual Text-to-Text LLM for The Medical Domain](https://mlsanity.com/paper/2404.07613/) (2024)
- [Beyond Fine-tuning: Unleashing the Potential of Continuous Pretraining for Clinical LLMs](https://mlsanity.com/paper/2409.14988/) (2024)
- [MORQA: Benchmarking Evaluation Metrics for Medical Open-Ended Question Answering](https://mlsanity.com/paper/2509.12405/) (2025)
- [MediQAl: A French Medical Question Answering Dataset for Knowledge and Reasoning Evaluation](https://mlsanity.com/paper/2507.20917/) (2025)
- [Who Judges the Judge? Evaluating LLM-as-a-Judge for French Medical open-ended QA](https://mlsanity.com/paper/2603.04033/) (2026)
- [MedInjection-FR: Exploring the Role of Native, Synthetic, and Translated Data in Biomedical Instruction Tuning](https://mlsanity.com/paper/2603.06905/) (2026)

## Abstract (arXiv)

The development of large language models (LLMs) has led to an increased focus on their adaptation to specialized domains and languages, yet the effectiveness of domain adaptation strategies remains unclear. We present a study of medical domain adaptation using French medical question-answering (QA) as a case study. We compare continual pretraining (CPT), supervised fine-tuning (SFT), and their combination across three model families, multiple sizes, and three initialization types, explicitly disentangling adaptation effects from base model choice. We evaluate both multiple-choice (MCQA) and open-ended QA (OEQA) under greedy and constrained decoding using automatic metrics and LLM-as-a-Judge evaluation. For MCQA, CPT+SFT most often achieves the best scores, but gains over SFT are small and frequently not statistically significant, making SFT a strong and cost-effective default. For OEQA, CPT consistently improves overlap-based metrics, while SFT often degrades generation quality; instruction tuning and CPT+SFT are preferred by LLM-based evaluation. Cross-lingual experiments further show effective transfer from French adaptation to English benchmarks. Overall, we provide practical guidelines for selecting adaptation strategies under computational constraints.