# Probabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Order

arXiv: https://arxiv.org/abs/2004.11579 · HTML: https://mlsanity.com/paper/2004.11579/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Apr 2020.

## Built on by (most cited first, 28 of 28)

- [Simple and Effective Masked Diffusion Language Models](https://mlsanity.com/paper/2406.07524/) (2024)
- [Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions](https://mlsanity.com/paper/2502.06768/) (2025)
- [Scene Text Recognition with Permuted Autoregressive Sequence Models](https://mlsanity.com/paper/2207.06966/) (2022)
- [Should You Mask 15% in Masked Language Modeling?](https://mlsanity.com/paper/2202.08005/) (2022)
- [Fighting Fire with Fire: Can ChatGPT Detect AI-generated Text?](https://mlsanity.com/paper/2308.01284/) (2023)
- [M6-UFC: Unifying Multi-Modal Controls for Conditional Image Synthesis via Non-Autoregressive Generative Transformers](https://mlsanity.com/paper/2105.14211/) (2021)
- [Sensitivity as a Complexity Measure for Sequence Classification Tasks](https://mlsanity.com/paper/2104.10343/) (2021)
- [Recent Advances in Neural Text Generation: A Task-Agnostic Survey](https://mlsanity.com/paper/2203.03047/) (2022)
- [House-GAN++: Generative Adversarial Layout Refinement Networks](https://mlsanity.com/paper/2103.02574/) (2021)
- [A New Approach to Overgenerating and Scoring Abstractive Summaries](https://mlsanity.com/paper/2104.01726/) (2021)
- [FiLM: Fill-in Language Models for Any-Order Generation](https://mlsanity.com/paper/2310.09930/) (2023)
- [IGA : An Intent-Guided Authoring Assistant](https://mlsanity.com/paper/2104.07000/) (2021)
- [BERT-based distractor generation for Swedish reading comprehension questions using a small-scale dataset](https://mlsanity.com/paper/2108.03973/) (2021)
- [Unsupervised Text Deidentification](https://mlsanity.com/paper/2210.11528/) (2022)
- [The Mystery of the Pathological Path-star Task for Language Models](https://mlsanity.com/paper/2410.13779/) (2024)
- [What Averages Do Not Tell -- Predicting Real Life Processes with Sequential Deep Learning](https://mlsanity.com/paper/2110.10225/) (2021)
- [Elucidating the design space of language models for image generation](https://mlsanity.com/paper/2410.16257/) (2024)
- [Unmasking Trees for Tabular Data](https://mlsanity.com/paper/2407.05593/) (2024)
- [Quinductor: a multilingual data-driven method for generating reading-comprehension questions using Universal Dependencies](https://mlsanity.com/paper/2103.10121/) (2021)
- [M6-Fashion: High-Fidelity Multi-modal Image Generation and Editing](https://mlsanity.com/paper/2205.11705/) (2022)
- [BLens: Contrastive Captioning of Binary Functions using Ensemble Embedding](https://mlsanity.com/paper/2409.07889/) (2024)
- [Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More More](https://mlsanity.com/paper/2503.10542/) (2025)
- [Fusing Sentence Embeddings Into LSTM-based Autoregressive Language Models](https://mlsanity.com/paper/2208.02402/) (2022)
- [Transfer training from smaller language model](https://mlsanity.com/paper/2104.11390/) (2021)
- [Learning and Analyzing Generation Order for Undirected Sequence Models](https://mlsanity.com/paper/2112.09097/) (2021)
- [Automatically generating question-answer pairs for assessing basic reading comprehension in Swedish](https://mlsanity.com/paper/2211.15568/) (2022)
- [Improving Pre-trained Language Model Sensitivity via Mask Specific losses: A case study on Biomedical NER](https://mlsanity.com/paper/2403.18025/) (2024)
- [POSPAN: Position-Constrained Span Masking for Language Model Pre-training](https://mlsanity.com/paper/2609.16061/) (2026)

## Abstract (arXiv)

Masked language model and autoregressive language model are two types of language models. While pretrained masked language models such as BERT overwhelm the line of natural language understanding (NLU) tasks, autoregressive language models such as GPT are especially capable in natural language generation (NLG). In this paper, we propose a probabilistic masking scheme for the masked language model, which we call probabilistically masked language model (PMLM). We implement a specific PMLM with a uniform prior distribution on the masking ratio named u-PMLM. We prove that u-PMLM is equivalent to an autoregressive permutated language model. One main advantage of the model is that it supports text generation in arbitrary order with surprisingly good quality, which could potentially enable new applications over traditional unidirectional generation. Besides, the pretrained u-PMLM also outperforms BERT on a set of downstream NLU tasks.