# Masked Language Model Scoring

arXiv: https://arxiv.org/abs/1910.14659 · HTML: https://mlsanity.com/paper/1910.14659/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Oct 2019.

## Built on by (most cited first, 4 of 4)

- [Diffusion on language model encodings for protein sequence generation](https://mlsanity.com/paper/2403.03726/) (2024)
- [Large Language Models as Mirrors of Societal Moral Standards](https://mlsanity.com/paper/2412.00956/) (2024)
- [Exploring Cultural Variations in Moral Judgments with Large Language Models](https://mlsanity.com/paper/2506.12433/) (2025)
- [How Can LLMs and Knowledge Graphs Contribute to Robot Safety? A Few-Shot Learning Approach](https://mlsanity.com/paper/2412.11387/) (2024)

## Abstract (arXiv)

Pretrained masked language models (MLMs) require finetuning for most NLP tasks. Instead, we evaluate MLMs out of the box via their pseudo-log-likelihood scores (PLLs), which are computed by masking tokens one by one. We show that PLLs outperform scores from autoregressive language models like GPT-2 in a variety of tasks. By rescoring ASR and NMT hypotheses, RoBERTa reduces an end-to-end LibriSpeech model's WER by 30% relative and adds up to +1.7 BLEU on state-of-the-art baselines for low-resource translation pairs, with further gains from domain adaptation. We attribute this success to PLL's unsupervised expression of linguistic acceptability without a left-to-right bias, greatly improving on scores from GPT-2 (+10 points on island effects, NPI licensing in BLiMP). One can finetune MLMs to give scores without masking, enabling computation in a single inference pass. In all, PLLs and their associated pseudo-perplexities (PPPLs) enable plug-and-play use of the growing number of pretrained MLMs; e.g., we use a single cross-lingual model to rescore translations in multiple languages. We release our library for language model scoring at https://github.com/awslabs/mlm-scoring.