mlsanityData to 14 Oct 2026

DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domains

arXiv 2304.00958 ↗First posted 3 Apr 2023cs.CLcited by 27

Not on the map. Why?

Abstract (arXiv)

In recent years, pre-trained language models (PLMs) achieve the best performance on a wide range of natural language processing (NLP) tasks. While the first models were trained on general domain data, specialized ones have emerged to more effectively treat specific domains. In this paper, we propose an original study of PLMs in the medical domain on French language. We compare, for the first time, the performance of PLMs trained on both public data from the web and private data from healthcare establishments. We also evaluate different learning strategies on a set of biomedical tasks. In particular, we show that we can take advantage of already existing biomedical PLMs in a foreign language by further pre-train it on our targeted data. Finally, we release the first specialized PLMs for the biomedical field in French, called DrBERT, as well as the largest corpus of medical data under free license on which these models are trained.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Builds on 22 of its 44 references that are in the map. First posted on arXiv 3 Apr 2023.

Builds on

22 of 44 references.

References · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  5. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  7. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  8. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  9. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  10. 2017Mixed Precision Trainingcited by 1,156
  11. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  12. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 493
10 more
  1. 2019CamemBERT: a Tasty French Language Modelcited by 368
  2. 2019Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasetscited by 299
  3. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 265
  4. 2022GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Recordscited by 238
  5. 2021Societal Biases in Language Generation: Progress and Challengescited by 202
  6. 2020FinBERT: A Pretrained Language Model for Financial Communicationscited by 161
  7. 2022MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languagescited by 146
  8. 2020When Do You Need Billions of Words of Pretraining Data?cited by 104
  9. 2021Pre-training Language Model Incorporating Domain-specific Heterogeneous Knowledge into A Unified Representationcited by 11
  10. 2021Biomedical and Clinical Language Models for Spanish: On the Benefits of Domain-Specific Pretraining in a Mid-Resource Scenariocited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 15; 2025: 1; 2026: 6.

27 papers.

Sort:

By area:

  1. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 301
  2. 2024Few-shot clinical entity recognition in English, French and Spanish: masked language models outperform generative model promptingcited by 12
  3. 2024DrBenchmark: A Large Language Understanding Evaluation Benchmark for French Biomedical Domaincited by 11
  4. 2023CamemBERT-bio: Leveraging Continual Pre-training for Cost-Effective Models on French Biomedical Datacited by 7
  5. 2023Multilingual Clinical NER: Translation or Cross-lingual Transfer?cited by 6
  6. 2024The Impact of LoRA Adapters on LLMs for Clinical Text Classification Under Computational and Data Constraintscited by 4
  7. 2024Conversational Medical AI: Ready for Practicecited by 4
  8. 2023xMEN: A Modular Toolkit for Cross-Lingual Medical Entity Normalizationcited by 3
  9. 2024Named Entity Recognition Under Domain Shift via Metric Learning for Life Sciencescited by 2
  10. 2024Language Model Adaptation to Specialized Domains through Selective Masking based on Genre and Topical Characteristicscited by 2
  11. 2024Adaptation of Biomedical and Clinical Pretrained Models to French Long Documents: A Comparative Studycited by 2
  12. 2023UMLS-KGI-BERT: Data-Centric Knowledge Integration in Transformers for Biomedical Entity Recognitioncited by 1
15 more
  1. 2024Chem-FINESE: Validating Fine-Grained Few-shot Entity Extraction through Text Reconstructioncited by 1
  2. 2024A Benchmark Evaluation of Clinical Named Entity Recognition in Frenchcited by 1
  3. 2026LLM-Based Data Generation and Clinical Skills Evaluation for Low-Resource French OSCEscited by 1
  4. 2026A Causal Language Modeling Detour Improves Encoder Continued Pretrainingcited by 1
  5. 2023A Survey of Spanish Clinical Language Modelscited by 0
  6. 2024Fine-Tuned Large Language Models for Symptom Recognition from Spanish Clinical Textcited by 0
  7. 2024How Important Is Tokenization in French Medical Masked Language Models?cited by 0
  8. 2024Comprehensive Study on German Language Models for Clinical and Biomedical Text Understandingcited by 0
  9. 2024Self-supervised vision-langage alignment of deep learning representations for bone X-rays analysiscited by 0
  10. 2024Retrieve, Generate, Evaluate: A Case Study for Medical Paraphrases Generation with Small Language Modelscited by 0
  11. 2025Towards the Anonymization of the Language Modelingcited by 0
  12. 2026Is Biomedical Specialization Still Worth It? Insights from Domain-Adaptive Language Modelling with a New French Health Corpuscited by 0
  13. 2026Trade-offs in Medical LLM Adaptation: An Empirical Study in French QAcited by 0
  14. 2026Where Does the Signal Live? A Web Data Recipe for Medical Encoder Pretrainingcited by 0
  15. 2026OntoBook: Ontology-Grounded Synthetic Textbooks for Medical Encoder Pretrainingcited by 0