mlsanityData to 14 Oct 2026

BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLP

arXiv 2506.10896 ↗First posted 12 Jun 2025cs.CL cs.AIcited by 28

Not on the map. Why?

Abstract (arXiv)

Encoder-based transformer models are central to biomedical and clinical Natural Language Processing (NLP), as their bidirectional self-attention makes them well-suited for efficiently extracting structured information from unstructured text through discriminative tasks. However, encoders have seen slower development compared to decoder models, leading to limited domain adaptation in biomedical and clinical settings. We introduce BioClinical ModernBERT, a domain-adapted encoder that builds on the recent ModernBERT release, incorporating long-context processing and substantial improvements in speed and performance for biomedical and clinical NLP. BioClinical ModernBERT is developed through continued pretraining on the largest biomedical and clinical corpus to date, with over 53.5 billion tokens, and addresses a key limitation of prior clinical encoders by leveraging 20 datasets from diverse institutions, domains, and geographic regions, rather than relying on data from a single source. It outperforms existing biomedical and clinical encoders on four downstream tasks spanning a broad range of use cases. We release both base (150M parameters) and large (396M parameters) versions of BioClinical ModernBERT, along with training checkpoints to support further research.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 30 of its 67 references that are in the map. First posted on arXiv 12 Jun 2025.

Builds on

30 of 67 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2020Longformer: The Long-Document Transformercited by 3,413
  7. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  8. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  9. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  10. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  11. 2019Publicly Available Clinical BERT Embeddingscited by 1,002
  12. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
18 more
  1. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 578
  2. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
  3. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 463
  4. 2019Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasetscited by 299
  5. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
  6. 2018Lessons from Natural Language Inference in the Clinical Domaincited by 151
  7. 2023The Semantic Scholar Open Data Platformcited by 118
  8. 2023ACI-BENCH: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generationcited by 118
  9. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  10. 2024Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspectivecited by 81
  11. 2025JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Modelscited by 71
  12. 2022Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequencescited by 69
  13. 2023MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretrainingcited by 34
  14. 2024Encoder vs Decoder: Comparative Analysis of Encoder and Decoder Language Models on Multilingual NLU Taskscited by 34
  15. 2025Clinical ModernBERT: An efficient and long context encoder for biomedical textcited by 26
  16. 2021Q-Pain: A Question Answering Dataset to Measure Social Bias in Pain Managementcited by 16
  17. 2023Dynamic Masking Rate Schedules for MLM Pretrainingcited by 12
  18. 2020A Corpus for Detecting High-Context Medical Conditions in Intensive Care Patient Notes Focusing on Frequently Readmitted Patientscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 20.

28 papers.

Sort:

By area:

  1. 2025RadEval: A framework for radiology text evaluationcited by 22
  2. 2026From Conflict to Consensus: Boosting Medical Reasoning via Multi-Round Agentic RAGcited by 7
  3. 2025Rethinking Tokenization for Clinical Time Series: When Less is Morecited by 3
  4. 2025Identifying Imaging Follow-Up in Radiology Reports: A Comparative Analysis of Traditional ML and LLM Approachescited by 1
  5. 2025Generalised Medical Phrase Groundingcited by 1
  6. 2025TabiBERT: A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkishcited by 1
  7. 2026SHIELD: A Diverse Clinical Note Dataset and Distilled Small Language Models for Enterprise-Scale De-identificationcited by 1
  8. 2026ShifaMind: A Multiplicative Concept Bottleneck for Interpretable ICD-10 Codingcited by 1
  9. 2026A Causal Language Modeling Detour Improves Encoder Continued Pretrainingcited by 1
  10. 2025Deep learning and abstractive summarisation for radiological reports: an empirical study for adapting the PEGASUS models' family with scarce datacited by 0
  11. 2025Automated Identification of Incidentalomas Requiring Follow-Up: A Multi-Anatomy Evaluation of LLM-Based and Supervised Approachescited by 0
  12. 2025SwinTF3D: A Lightweight Multimodal Fusion Approach for Text-Guided 3D Medical Image Segmentationcited by 0
16 more
  1. 2026Predicting first-episode homelessness among US Veterans using longitudinal EHR data: time-varying models and social risk factorscited by 0
  2. 2026RexBERT: Context Specialized Bidirectional Encoders for E-commercecited by 0
  3. 2026MrBERT: Modern Multilingual Encoders via Vocabulary, Domain, and Dimensional Adaptationcited by 0
  4. 2026Improving Pediatric Emergency Department Triage with Modality Dropout in Late Fusion Multimodal EHR Modelscited by 0
  5. 2026Evidence-Grounded Frontier Mapping and Agentic Hypothesis Generation in Nanomedicinecited by 0
  6. 2026Learning to Select Source-Traceable Evidence for Language-Model Prediction from Irregular Clinical Time Seriescited by 0
  7. 2026Where Does the Signal Live? A Web Data Recipe for Medical Encoder Pretrainingcited by 0
  8. 2026How much of an LLM-generated clinical corpus is actually new? A production-scale measurement of content redundancy for provenance classificationcited by 0
  9. 2026Patients-like-me: A Variational LM--GNN Framework for Explainable Clinical Predictioncited by 0
  10. 2026Mr.Dec: Daily-Scale Longitudinal Multimodal Modeling for 30-Day Readmission Predictioncited by 0
  11. 2026Institution-Specific LLM Prompting Recovers PHI That De-identification Systems and Their Gold Standards Both Misscited by 0
  12. 2026MorphoCLIP: Text-Supervised Contrastive Learning for Perturbation Matching in Cell Painting Imagescited by 0
  13. 2026Climate-ModernBERT: Revisiting Corpus Composition for Domain-Adaptive Continued Pretrainingcited by 0
  14. 2026A Multimodal Dataset for Survival Prediction in Resected Pancreatic Ductal Adenocarcinomacited by 0
  15. 2026From Retrieval to Typed Decisions: Calibrated System One Models from Biomedical Sentence Encoderscited by 0
  16. 2026TIDE 2.0: an open, model-agnostic engine for keyed de-identification of clinical notescited by 0