# RuBioRoBERTa: a pre-trained biomedical language model for Russian language biomedical text mining

arXiv: https://arxiv.org/abs/2204.03951 · HTML: https://mlsanity.com/paper/2204.03951/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 7 references that are in the map.
- First posted on arXiv 8 Apr 2022.

## Builds on (most cited first, 7 of 7)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets](https://mlsanity.com/paper/1906.05474/) (2019)
- [Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language](https://mlsanity.com/paper/1905.07213/) (2019)
- [RuMedBench: A Russian Medical Language Understanding Benchmark](https://mlsanity.com/paper/2201.06499/) (2022)

## Built on by (most cited first, 6 of 6)

- [MedSyn: LLM-based Synthetic Medical Text Generation Framework](https://mlsanity.com/paper/2408.02056/) (2024)
- [Uncertainty-aware abstention in medical diagnosis based on medical texts](https://mlsanity.com/paper/2502.18050/) (2025)
- [Exploring Large Language Models for Detecting Mental Disorders](https://mlsanity.com/paper/2410.07129/) (2024)
- [RuCCoD: Towards Automated ICD Coding in Russian](https://mlsanity.com/paper/2502.21263/) (2025)
- [Can-SAVE: Deploying Low-Cost and Population-Scale Cancer Screening via Survival Analysis Variables and EHR](https://mlsanity.com/paper/2309.15039/) (2023)
- [Pre-AF 13: An Interpretable Atrial Fibrillation Risk Score Mined from Discharge Reports](https://mlsanity.com/paper/2606.10725/) (2026)

## Abstract (arXiv)

This paper presents several BERT-based models for Russian language biomedical text mining (RuBioBERT, RuBioRoBERTa). The models are pre-trained on a corpus of freely available texts in the Russian biomedical domain. With this pre-training, our models demonstrate state-of-the-art results on RuMedBench - Russian medical language understanding benchmark that covers a diverse set of tasks, including text classification, question answering, natural language inference, and named entity recognition.