# Deriving Language Models from Masked Language Models

arXiv: https://arxiv.org/abs/2305.15501 · HTML: https://mlsanity.com/paper/2305.15501/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 33 references that are in the map.
- First posted on arXiv 24 May 2023.

## Builds on (most cited first, 11 of 11)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model](https://mlsanity.com/paper/1902.04094/) (2019)
- [Generalized Denoising Auto-Encoders as Generative Models](https://mlsanity.com/paper/1305.6663/) (2013)
- [Exposing the Implicit Energy Networks behind Masked Language Models via Metropolis--Hastings](https://mlsanity.com/paper/2106.02736/) (2021)
- [Deep Generative Stochastic Networks Trainable by Backprop](https://mlsanity.com/paper/1306.1091/) (2013)
- [A Measure-Theoretic Characterization of Tight Language Models](https://mlsanity.com/paper/2212.10502/) (2022)
- [Probing BERT's priors with serial reproduction chains](https://mlsanity.com/paper/2202.12226/) (2022)

## Built on by (most cited first, 9 of 9)

- [Amortizing intractable inference in large language models](https://mlsanity.com/paper/2310.04363/) (2023)
- [Promises and Pitfalls of Generative Masked Language Modeling: Theoretical Framework and Practical Guidelines](https://mlsanity.com/paper/2407.21046/) (2024)
- [Masked Language Modeling Becomes Conditional Density Estimation for Tabular Data Synthesis](https://mlsanity.com/paper/2405.20602/) (2024)
- [Generative AI & Fictionality: How Novels Power Large Language Models](https://mlsanity.com/paper/2603.01220/) (2026)
- [Towards Probabilistically-Sound Beam Search with Masked Language Models](https://mlsanity.com/paper/2402.15020/) (2024)
- [One protein is all you need](https://mlsanity.com/paper/2411.02109/) (2024)
- [Learning Extrapolative Sequence Transformations from Markov Chains](https://mlsanity.com/paper/2505.20251/) (2025)
- [Structured Inference with Large Language Gibbs](https://mlsanity.com/paper/2606.19264/) (2026)
- [Nash Equilibrium Text: A Game-Theoretic Decoding Framework for Text Generation](https://mlsanity.com/paper/2610.05817/) (2026)

## Abstract (arXiv)

Masked language models (MLM) do not explicitly define a distribution over language, i.e., they are not language models per se. However, recent work has implicitly treated them as such for the purposes of generation and scoring. This paper studies methods for deriving explicit joint distributions from MLMs, focusing on distributions over two tokens, which makes it possible to calculate exact distributional properties. We find that an approach based on identifying joints whose conditionals are closest to those of the MLM works well and outperforms existing Markov random field-based approaches. We further find that this derived model's conditionals can even occasionally outperform the original MLM's conditionals.