# DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domains

arXiv: https://arxiv.org/abs/2304.00958 · HTML: https://mlsanity.com/paper/2304.00958/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 44 references that are in the map.
- First posted on arXiv 3 Apr 2023.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission](https://mlsanity.com/paper/1904.05342/) (2019)
- [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019)
- [Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets](https://mlsanity.com/paper/1906.05474/) (2019)
- [Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model](https://mlsanity.com/paper/2211.02001/) (2022)
- [GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Records](https://mlsanity.com/paper/2203.03540/) (2022)
- [Societal Biases in Language Generation: Progress and Challenges](https://mlsanity.com/paper/2105.04054/) (2021)
- [FinBERT: A Pretrained Language Model for Financial Communications](https://mlsanity.com/paper/2006.08097/) (2020)
- [MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages](https://mlsanity.com/paper/2204.08582/) (2022)
- [When Do You Need Billions of Words of Pretraining Data?](https://mlsanity.com/paper/2011.04946/) (2020)
- [Pre-training Language Model Incorporating Domain-specific Heterogeneous Knowledge into A Unified Representation](https://mlsanity.com/paper/2109.01048/) (2021)
- [Biomedical and Clinical Language Models for Spanish: On the Benefits of Domain-Specific Pretraining in a Mid-Resource Scenario](https://mlsanity.com/paper/2109.03570/) (2021)

## Built on by (most cited first, 27 of 27)

- [BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains](https://mlsanity.com/paper/2402.10373/) (2024)
- [Few-shot clinical entity recognition in English, French and Spanish: masked language models outperform generative model prompting](https://mlsanity.com/paper/2402.12801/) (2024)
- [DrBenchmark: A Large Language Understanding Evaluation Benchmark for French Biomedical Domain](https://mlsanity.com/paper/2402.13432/) (2024)
- [CamemBERT-bio: Leveraging Continual Pre-training for Cost-Effective Models on French Biomedical Data](https://mlsanity.com/paper/2306.15550/) (2023)
- [Multilingual Clinical NER: Translation or Cross-lingual Transfer?](https://mlsanity.com/paper/2306.04384/) (2023)
- [The Impact of LoRA Adapters on LLMs for Clinical Text Classification Under Computational and Data Constraints](https://mlsanity.com/paper/2407.19299/) (2024)
- [Conversational Medical AI: Ready for Practice](https://mlsanity.com/paper/2411.12808/) (2024)
- [xMEN: A Modular Toolkit for Cross-Lingual Medical Entity Normalization](https://mlsanity.com/paper/2310.11275/) (2023)
- [Named Entity Recognition Under Domain Shift via Metric Learning for Life Sciences](https://mlsanity.com/paper/2401.10472/) (2024)
- [Language Model Adaptation to Specialized Domains through Selective Masking based on Genre and Topical Characteristics](https://mlsanity.com/paper/2402.12036/) (2024)
- [Adaptation of Biomedical and Clinical Pretrained Models to French Long Documents: A Comparative Study](https://mlsanity.com/paper/2402.16689/) (2024)
- [UMLS-KGI-BERT: Data-Centric Knowledge Integration in Transformers for Biomedical Entity Recognition](https://mlsanity.com/paper/2307.11170/) (2023)
- [Chem-FINESE: Validating Fine-Grained Few-shot Entity Extraction through Text Reconstruction](https://mlsanity.com/paper/2401.10189/) (2024)
- [A Benchmark Evaluation of Clinical Named Entity Recognition in French](https://mlsanity.com/paper/2403.19726/) (2024)
- [LLM-Based Data Generation and Clinical Skills Evaluation for Low-Resource French OSCEs](https://mlsanity.com/paper/2604.08126/) (2026)
- [A Causal Language Modeling Detour Improves Encoder Continued Pretraining](https://mlsanity.com/paper/2605.12438/) (2026)
- [A Survey of Spanish Clinical Language Models](https://mlsanity.com/paper/2308.02199/) (2023)
- [Fine-Tuned Large Language Models for Symptom Recognition from Spanish Clinical Text](https://mlsanity.com/paper/2401.15780/) (2024)
- [How Important Is Tokenization in French Medical Masked Language Models?](https://mlsanity.com/paper/2402.15010/) (2024)
- [Comprehensive Study on German Language Models for Clinical and Biomedical Text Understanding](https://mlsanity.com/paper/2404.05694/) (2024)
- [Self-supervised vision-langage alignment of deep learning representations for bone X-rays analysis](https://mlsanity.com/paper/2405.08932/) (2024)
- [Retrieve, Generate, Evaluate: A Case Study for Medical Paraphrases Generation with Small Language Models](https://mlsanity.com/paper/2407.16565/) (2024)
- [Towards the Anonymization of the Language Modeling](https://mlsanity.com/paper/2501.02407/) (2025)
- [Is Biomedical Specialization Still Worth It? Insights from Domain-Adaptive Language Modelling with a New French Health Corpus](https://mlsanity.com/paper/2604.06903/) (2026)
- [Trade-offs in Medical LLM Adaptation: An Empirical Study in French QA](https://mlsanity.com/paper/2606.19266/) (2026)
- [Where Does the Signal Live? A Web Data Recipe for Medical Encoder Pretraining](https://mlsanity.com/paper/2606.22079/) (2026)
- [OntoBook: Ontology-Grounded Synthetic Textbooks for Medical Encoder Pretraining](https://mlsanity.com/paper/2607.18927/) (2026)

## Abstract (arXiv)

In recent years, pre-trained language models (PLMs) achieve the best performance on a wide range of natural language processing (NLP) tasks. While the first models were trained on general domain data, specialized ones have emerged to more effectively treat specific domains. In this paper, we propose an original study of PLMs in the medical domain on French language. We compare, for the first time, the performance of PLMs trained on both public data from the web and private data from healthcare establishments. We also evaluate different learning strategies on a set of biomedical tasks. In particular, we show that we can take advantage of already existing biomedical PLMs in a foreign language by further pre-train it on our targeted data. Finally, we release the first specialized PLMs for the biomedical field in French, called DrBERT, as well as the largest corpus of medical data under free license on which these models are trained.