mlsanityData to 14 Oct 2026

Clinical ModernBERT: An efficient and long context encoder for biomedical text

arXiv 2504.03964 ↗First posted 4 Apr 2025cs.CL cs.AI cs.LGcited by 26

Not on the map. Why?

Abstract (arXiv)

We introduce Clinical ModernBERT, a transformer based encoder pretrained on large scale biomedical literature, clinical notes, and medical ontologies, incorporating PubMed abstracts, MIMIC IV clinical data, and medical codes with their textual descriptions. Building on ModernBERT the current state of the art natural language text encoder featuring architectural upgrades such as rotary positional embeddings (RoPE), Flash Attention, and extended context length up to 8,192 tokens our model adapts these innovations specifically for biomedical and clinical domains. Clinical ModernBERT excels at producing semantically rich representations tailored for long context tasks. We validate this both by analyzing its pretrained weights and through empirical evaluation on a comprehensive suite of clinical NLP benchmarks.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 31 of its 43 references that are in the map. First posted on arXiv 4 Apr 2025.

Builds on

31 of 43 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2020Longformer: The Long-Document Transformercited by 3,413
  8. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  9. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  10. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  11. 2020GLU Variants Improve Transformercited by 1,804
  12. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
19 more
  1. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  2. 2019Publicly Available Clinical BERT Embeddingscited by 1,002
  3. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
  4. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 493
  5. 2019Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasetscited by 299
  6. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 277
  7. 2020Sparse Sinkhorn Attentioncited by 251
  8. 2020Self-Alignment Pretraining for Biomedical Entity Representationscited by 237
  9. 2019Text Classification Algorithms: A Surveycited by 163
  10. 2017PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstractscited by 124
  11. 2023Stable and low-precision training for large-scale vision-language modelscited by 75
  12. 2024Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHRscited by 39
  13. 2023MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretrainingcited by 34
  14. 2024Emergency Department Decision Support using Clinical Pseudo-notescited by 22
  15. 2024Can Large Language Models abstract Medical Coded Language?cited by 20
  16. 2024Text Serialization and Their Relationship with the Conventional Paradigms of Tabular Machine Learningcited by 17
  17. 2023An Empirical Analysis of Parameter-Efficient Methods for Debiasing Pre-Trained Language Modelscited by 16
  18. 2024Enhancing Antibiotic Stewardship using a Natural Language Approach for Better Feature Representationcited by 8
  19. 2025ModernBERT is More Efficient than Conventional BERT for Chest CT Findings Classification in Japanese Radiology Reportscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 17.

26 papers.

Sort:

By area:

  1. 2025Seq vs Seq: An Open Suite of Paired Encoders and Decoderscited by 38
  2. 2025BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLPcited by 28
  3. 2025A Case Study Exploring the Current Landscape of Synthetic Medical Record Generation with Commercial LLMscited by 13
  4. 2025ChronoFormer: Time-Aware Transformer Architectures for Structured Clinical Event Modelingcited by 12
  5. 2025Structured Semantics from Unstructured Notes: Language Model Approaches to EHR-Based Decision Supportcited by 7
  6. 2026GeMM-GAN: A Multimodal Generative Model Conditioned on Histopathology Images and Clinical Descriptions for Gene Expression Profile Generationcited by 2
  7. 2026Learning Clinical Representations Under Systematic Distribution Shiftcited by 2
  8. 2026Discriminative Representation Learning for Clinical Predictioncited by 2
  9. 2025Temporal Entailment Pretraining for Clinical Language Models over EHR Datacited by 1
  10. 2025SemCSE-Multi: Multifaceted and Decodable Embeddings for Aspect-Specific and Interpretable Scientific Domain Mappingcited by 1
  11. 2025TabiBERT: A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkishcited by 1
  12. 2026Learning Longitudinal Health Representations from EHR and Wearable Datacited by 1
14 more
  1. 2026Early Risk Stratification of Dosing Errors in Clinical Trials Using Machine Learningcited by 1
  2. 2026Uncertainty-Aware Foundation Models for Clinical Datacited by 1
  3. 2026A Causal Language Modeling Detour Improves Encoder Continued Pretrainingcited by 1
  4. 2026Combating Knowledge Corruption in Agent Systems: A Byzantine-Tolerant Secure Collaborative RAG Frameworkcited by 1
  5. 2025Hierarchical Dual-Strategy Unlearning for Biomedical and Healthcare Intelligence Using Imperfect and Privacy-Sensitive Medical Datacited by 0
  6. 2026Dense Feature Learning via Linear Structure Preservation in Medical Datacited by 0
  7. 2026PRIMA: Pre-Training with Risk-Integrated Image--Metadata Alignment for Medical Diagnosis with LLM-Based Feature Aggregationcited by 0
  8. 2026Event Fields: Learning Latent Event Structure for Waveform Foundation Modelscited by 0
  9. 2026WISTERIA: Learning Clinical Representations from Noisy Supervision via Multi-View Consistency in Electronic Health Recordscited by 0
  10. 2026AURORA: Contextual Orthogonalization for Geometric Representation Learning in Healthcare Foundation Modelscited by 0
  11. 2026Traj-Evolve: A Self-Evolving Multi-Agent System for Patient Trajectory Modeling in Lung Cancer Early Detectioncited by 0
  12. 2026Where Does the Signal Live? A Web Data Recipe for Medical Encoder Pretrainingcited by 0
  13. 2026MorphologyFM: A Foundation Model for Morphology-Aware Representation Learning from ECG and Pulse Oximetry Waveformscited by 0
  14. 2026M3-Gen: Interpretable Multimodal Generation of Gene Expression Profiles Using Clinical and Imaging Datacited by 0