mlsanityData to 14 Oct 2026

Masked Diffusion Language Models with Frequency-Informed Training

arXiv 2509.05056 ↗First posted 5 Sep 2025cs.CLcited by 2

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

We present a masked diffusion language modeling framework for data-efficient training for the BabyLM 2025 Challenge. Our approach applies diffusion training objectives to language modeling under strict data constraints, incorporating frequency-informed masking that prioritizes learning from rare tokens while maintaining theoretical validity. We explore multiple noise scheduling strategies, including two-mode approaches, and investigate different noise weighting schemes within the NELBO objective. We evaluate our method on the BabyLM benchmark suite, measuring linguistic competence, world knowledge, and human-likeness. Results show performance competitive to hybrid autoregressive-masked baselines, demonstrating that diffusion-based training offers a viable alternative for data-restricted language learning.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 27 of its 43 references that are in the map. First posted on arXiv 5 Sep 2025.

Builds on

27 of 43 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025The Diffusion Duality12 Jun
  2. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  3. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun

Other references · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  8. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  9. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  10. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 574
  11. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  12. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 451
12 more
  1. 2019Competence-based Curriculum Learning for Neural Machine Translationcited by 202
  2. 2024Rho-1: Not All Tokens Are What You Needcited by 133
  3. 2023Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpuscited by 62
  4. 2022SinDDM: A Single Image Denoising Diffusion Modelcited by 60
  5. 2024Elements of World Knowledge (EWoK): A Cognition-Inspired Framework for Evaluating Basic World Knowledge in Language Modelscited by 55
  6. 2020PMI-Masking: Principled masking of correlated spanscited by 54
  7. 2023Trained on 100 million words and still in shape: BERT meets British National Corpuscited by 39
  8. 2024GPT or BERT: why not both?cited by 27
  9. 2025BabyLM Turns 3: Call for papers for the 2025 BabyLM workshopcited by 24
  10. 2022Few-shot Image Generation with Diffusion Modelscited by 21
  11. 2023Not all layers are equally as important: Every Layer Counts BERTcited by 21
  12. 2024Salience-Based Adaptive Masking: Revisiting Token Dynamics for Enhanced Pre-trainingcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Tuning the Implicit Regularizer of Masked Diffusion Language Models: Enhancing Generalization via Insights from k-Paritycited by 1
  2. 2026A retrospective analysis on the use of LLMs to study infant syntax learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Understanding and Accelerating the Training of Masked Diffusion Language Models
  2. 2024Scaling up Masked Diffusion Models on Text
  3. 2026Machine Unlearning for Masked Diffusion Language Models
  4. 2026Improved Large Language Diffusion Models
  5. 2023A Cheaper and Better Diffusion Language Model with Soft-Masked Noise
  6. 2025Soft-Masked Diffusion Language Models
  7. 2026Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Models
  8. 2026Autoregressive vs. Masked Diffusion Language Models: A Controlled Comparison