mlsanityData to 7 Oct 2026

A Causal Language Modeling Detour Improves Encoder Continued Pretraining

arXiv 2605.12438 ↗First posted 12 May 2026cs.CL cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

When adapting an encoder to a new domain, the standard approach is to continue training with Masked Language Modeling (MLM). We show that temporarily switching to Causal Language Modeling (CLM) followed by a short MLM decay improves downstream performance. On biomedical texts with ModernBERT, this CLM detour outperforms MLM baselines trained on identical data and compute across 8 French and 11 English biomedical tasks, by +1.2-2.8pp and +0.3-0.8pp respectively, depending on model size. We investigate the reasons for these gains. We find that CLM's dense supervision impacts low transformer layers (0-7) far more than MLM does. Freezing low layers during CLM eliminates the downstream benefit; freezing mid layers preserves it. The representational changes persist through the MLM decay phase, even when it matches the CLM phase in length, and they scale with model capacity. We release ModernCamemBERT-bio and ModernBERT-bio as state-of-the-art biomedical encoders in Base and Large sizes.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 33 of its 55 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

33 of 55 references.

References · 33

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  4. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  5. 2025Gemma 3 Technical Reportcited by 1,686
  6. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  7. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  8. 2019Publicly Available Clinical BERT Embeddingscited by 995
  9. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  10. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  11. 2019CamemBERT: a Tasty French Language Modelcited by 368
  12. 2020What is being transferred in transfer learning?cited by 365
21 more
  1. 2023Continual Pre-training of Language Modelscited by 194
  2. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 160
  3. 2020What Happens To BERT Embeddings During Fine-tuning?cited by 113
  4. 2019What Would Elsa Do? Freezing Layers During Transformer Fine-Tuningcited by 87
  5. 2020Similarity Analysis of Contextual Word Representation Modelscited by 54
  6. 2024Towards Evaluating and Building Versatile Large Language Models for Medicinecited by 38
  7. 2025Seq vs Seq: An Open Suite of Paired Encoders and Decoderscited by 34
  8. 2023DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domainscited by 27
  9. 2025Clinical ModernBERT: An efficient and long context encoder for biomedical textcited by 26
  10. 2025ReasonMed: A 370K Multi-Agent Generated Dataset for Advancing Medical Reasoningcited by 26
  11. 2025BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLPcited by 26
  12. 2023The 2022 n2c2/UW Shared Task on Extracting Social Determinants of Healthcited by 21
  13. 2025A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignmentcited by 20
  14. 2025EHR-R1: A Reasoning-Enhanced Foundational Language Model for Electronic Health Record Analysiscited by 17
  15. 2025Should We Still Pretrain Encoders with Masked Language Modeling?cited by 15
  16. 2024DrBenchmark: A Large Language Understanding Evaluation Benchmark for French Biomedical Domaincited by 11
  17. 2025ModernBERT or DeBERTaV3? Examining Architecture and Data Influence on Transformer Encoder Models Performancecited by 11
  18. 2024AntLM: Bridging Causal and Masked Language Modelscited by 8
  19. 2023CamemBERT-bio: Leveraging Continual Pre-training for Cost-Effective Models on French Biomedical Datacited by 6
  20. 2025Biomed-Enriched: A Biomedical Dataset Enriched with LLMs for Pretraining and Extracting Rare and Hidden Contentcited by 2
  21. 2025FRACCO: A gold-standard annotated corpus of oncological entities with ICD-O-3.1 normalisationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Where Does the Signal Live? A Web Data Recipe for Medical Encoder Pretrainingcited by 0