mlsanityData to 7 Oct 2026

Deriving Language Models from Masked Language Models

arXiv 2305.15501 ↗First posted 24 May 2023cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

Masked language models (MLM) do not explicitly define a distribution over language, i.e., they are not language models per se. However, recent work has implicitly treated them as such for the purposes of generation and scoring. This paper studies methods for deriving explicit joint distributions from MLMs, focusing on distributions over two tokens, which makes it possible to calculate exact distributional properties. We find that an approach based on identifying joints whose conditionals are closest to those of the MLM works well and outperforms existing Markov random field-based approaches. We further find that this derived model's conditionals can even occasionally outperform the original MLM's conditionals.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 11 of its 33 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

11 of 33 references.

References · 11

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  4. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  5. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  6. 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 184
  7. 2013Generalized Denoising Auto-Encoders as Generative Modelscited by 113
  8. 2021Exposing the Implicit Energy Networks behind Masked Language Models via Metropolis--Hastingscited by 37
  9. 2013Deep Generative Stochastic Networks Trainable by Backpropcited by 36
  10. 2022A Measure-Theoretic Characterization of Tight Language Modelscited by 32
  11. 2022Probing BERT's priors with serial reproduction chainscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 1; 2026: 2.

8 papers.

  1. 2023Amortizing intractable inference in large language modelscited by 94
  2. 2024Promises and Pitfalls of Generative Masked Language Modeling: Theoretical Framework and Practical Guidelinescited by 6
  3. 2024Masked Language Modeling Becomes Conditional Density Estimation for Tabular Data Synthesiscited by 1
  4. 2026Generative AI & Fictionality: How Novels Power Large Language Modelscited by 1
  5. 2024Towards Probabilistically-Sound Beam Search with Masked Language Modelscited by 0
  6. 2024One protein is all you needcited by 0
  7. 2025Learning Extrapolative Sequence Transformations from Markov Chainscited by 0
  8. 2026Structured Inference with Large Language Gibbscited by 0