# MEDBERT.de: A Comprehensive German BERT Model for the Medical Domain

arXiv: https://arxiv.org/abs/2303.08179 · HTML: https://mlsanity.com/paper/2303.08179/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 29 references that are in the map.
- First posted on arXiv 14 Mar 2023.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://mlsanity.com/paper/1904.00962/) (2019)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019)
- [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://mlsanity.com/paper/2012.15613/) (2020)
- [German's Next Language Model](https://mlsanity.com/paper/2010.10906/) (2020)
- [A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages](https://mlsanity.com/paper/2006.06202/) (2020)
- [GottBERT: a pure German Language Model](https://mlsanity.com/paper/2012.02110/) (2020)
- [How much pretraining data do language models need to learn syntax?](https://mlsanity.com/paper/2109.03160/) (2021)
- [GGPONC: A Corpus of German Medical Text with Rich Metadata Based on Clinical Practice Guidelines](https://mlsanity.com/paper/2007.06400/) (2020)

## Built on by (most cited first, 10 of 10)

- [Multilingual Clinical NER: Translation or Cross-lingual Transfer?](https://mlsanity.com/paper/2306.04384/) (2023)
- [On the Impact of Cross-Domain Data on German Language Models](https://mlsanity.com/paper/2310.07321/) (2023)
- [CareerBERT: Matching Resumes to ESCO Jobs in a Shared Embedding Space for Generic Job Recommendations](https://mlsanity.com/paper/2503.02056/) (2025)
- [xMEN: A Modular Toolkit for Cross-Lingual Medical Entity Normalization](https://mlsanity.com/paper/2310.11275/) (2023)
- [G-SciEdBERT: A Contextualized LLM for Science Assessment Tasks in German](https://mlsanity.com/paper/2402.06584/) (2024)
- [Clinical information extraction for Low-resource languages with Few-shot learning using Pre-trained language models and Prompting](https://mlsanity.com/paper/2403.13369/) (2024)
- [Can open source large language models be used for tumor documentation in Germany? -- An evaluation on urological doctors' notes](https://mlsanity.com/paper/2501.12106/) (2025)
- [Comprehensive Study on German Language Models for Clinical and Biomedical Text Understanding](https://mlsanity.com/paper/2404.05694/) (2024)
- [MedicalBERT: enhancing biomedical natural language processing using pretrained BERT-based model](https://mlsanity.com/paper/2507.08013/) (2025)
- [The Word and the Way: Strategies for Domain-Specific BERT Pre-Training in German Medical NLP](https://mlsanity.com/paper/2606.03250/) (2026)

## Abstract (arXiv)

This paper presents medBERTde, a pre-trained German BERT model specifically designed for the German medical domain. The model has been trained on a large corpus of 4.7 Million German medical documents and has been shown to achieve new state-of-the-art performance on eight different medical benchmarks covering a wide range of disciplines and medical document types. In addition to evaluating the overall performance of the model, this paper also conducts a more in-depth analysis of its capabilities. We investigate the impact of data deduplication on the model's performance, as well as the potential benefits of using more efficient tokenization methods. Our results indicate that domain-specific models such as medBERTde are particularly useful for longer texts, and that deduplication of training data does not necessarily lead to improved performance. Furthermore, we found that efficient tokenization plays only a minor role in improving model performance, and attribute most of the improved performance to the large amount of training data. To encourage further research, the pre-trained model weights and new benchmarks based on radiological data are made publicly available for use by the scientific community.