# Dynamic Masking Rate Schedules for MLM Pretraining

arXiv: https://arxiv.org/abs/2305.15096 · HTML: https://mlsanity.com/paper/2305.15096/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 43 references that are in the map.
- First posted on arXiv 24 May 2023.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Cyclical Learning Rates for Training Neural Networks](https://mlsanity.com/paper/1506.01186/) (2015)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [Don't Decay the Learning Rate, Increase the Batch Size](https://mlsanity.com/paper/1711.00489/) (2017)
- [Should You Mask 15% in Masked Language Modeling?](https://mlsanity.com/paper/2202.08005/) (2022)
- [PMI-Masking: Principled masking of correlated spans](https://mlsanity.com/paper/2010.01825/) (2020)
- [Curriculum Dropout](https://mlsanity.com/paper/1703.06229/) (2017)
- [The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Models](https://mlsanity.com/paper/2108.06084/) (2021)
- [Scheduled DropHead: A Regularization Method for Transformer Models](https://mlsanity.com/paper/2004.13342/) (2020)
- [Learning Better Masking for Better Language Model Pre-training](https://mlsanity.com/paper/2208.10806/) (2022)
- [Word Order Matters when you Increase Masking](https://mlsanity.com/paper/2211.04427/) (2022)
- [Effective Pre-Training Objectives for Transformer-based Autoencoders](https://mlsanity.com/paper/2210.13536/) (2022)

## Built on by (most cited first, 12 of 12)

- [MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining](https://mlsanity.com/paper/2312.17482/) (2023)
- [EuroBERT: Scaling Multilingual Encoders for European Languages](https://mlsanity.com/paper/2503.05500/) (2025)
- [BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLP](https://mlsanity.com/paper/2506.10896/) (2025)
- [GPT or BERT: why not both?](https://mlsanity.com/paper/2410.24159/) (2024)
- [Task-Informed Anti-Curriculum by Masking Improves Downstream Performance on Text](https://mlsanity.com/paper/2502.12953/) (2025)
- [Understanding and Enhancing Mask-Based Pretraining towards Universal Representations](https://mlsanity.com/paper/2509.21650/) (2025)
- [CLewR: Curriculum Learning with Restarts for Machine Translation Preference Learning](https://mlsanity.com/paper/2601.05858/) (2026)
- [DEPTH: Discourse Education through Pre-Training Hierarchically](https://mlsanity.com/paper/2405.07788/) (2024)
- [Chinese ModernBERT with Whole-Word Masking](https://mlsanity.com/paper/2510.12285/) (2025)
- [EVA: Towards a universal model of the immune system](https://mlsanity.com/paper/2602.10168/) (2026)
- [Polish ModernBERT: The Long and Short of Polish Language Understanding](https://mlsanity.com/paper/2609.01379/) (2026)
- [UltraBERT: Beyond the Limit of BERT Pre-Training](https://mlsanity.com/paper/2610.15028/) (2026)

## Abstract (arXiv)

Most works on transformers trained with the Masked Language Modeling (MLM) objective use the original BERT model's fixed masking rate of 15%. We propose to instead dynamically schedule the masking rate throughout training. We find that linearly decreasing the masking rate over the course of pretraining improves average GLUE accuracy by up to 0.46% and 0.25% in BERT-base and BERT-large, respectively, compared to fixed rate baselines. These gains come from exposure to both high and low masking rate regimes, providing benefits from both settings. Our results demonstrate that masking rate scheduling is a simple way to improve the quality of masked language models, achieving up to a 1.89x speedup in pretraining for BERT-base as well as a Pareto improvement for BERT-large.