Mask and You Shall Receive: Optimizing Masked Language Modeling For Pretraining BabyLMs
Not on the map. Why?
Abstract (arXiv)
We describe our strategy for the 2025 edition of the BabyLM Challenge. Our main contribution is that of an improved form of Masked Language Modeling (MLM), which adapts the probabilities of the tokens masked according to the model's ability to predict them. The results show a substantial increase in performance on (Super)GLUE tasks over the standard MLM. We also incorporate sub-token embeddings, finding that this increases the model's morphological generalization capabilities. Our submission beats the baseline in the strict-small track.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 11 of its 18 references that are in the map. First posted on arXiv 23 Oct 2025.
Builds on
11 of 18 references.
References · 11
- 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
- 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
- 2024Byte Latent Transformer: Patches Scale Better Than Tokenscited by 146
- 2025Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 88
- 2020CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characterscited by 87
- 2024Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 48
- 2025BabyLM Turns 3: Call for papers for the 2025 BabyLM workshopcited by 24
- 2024From Babble to Words: Pre-Training Language Models on Continuous Streams of Phonemescited by 11
- 2023Too Much Information: Keeping Training Simple for BabyLMscited by 7
- 2023Weighted Sampling for Masked Language Modelingcited by 2
- 2024Language Model Adaptation to Specialized Domains through Selective Masking based on Genre and Topical Characteristicscited by 2
Built on by
Citing papers per year: 2026: 3.
3 papers.