mlsanityData to 7 Oct 2026

Towards Probabilistically-Sound Beam Search with Masked Language Models

arXiv 2402.15020 ↗First posted 22 Feb 2024cs.LG cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Beam search with masked language models (MLMs) is challenging in part because joint probability distributions over sequences are not readily available, unlike for autoregressive models. However, estimating such distributions has important domain-specific applications such as ancient text restoration and protein engineering. Here we present probabilistically-sound methods for beam search with MLMs. First, we clarify the conditions under which it is theoretically sound to perform text infilling with MLMs using standard beam search. When these conditions fail, we provide a probabilistically-sound inference time modification with no additional computational complexity and demonstrate that it is superior to the aforementioned beam search in the expected conditions. We then present empirical results comparing several infilling approaches with MLMs across several domains. Notably, our method probes the inductive biases of MLMs and explores the surprising contextual sensitivity of mask tokens for text infilling.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 18 of its 35 references that are in the map. First posted on arXiv 22 Feb 2024.

Builds on

18 of 35 references.

References · 18

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  7. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  8. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  9. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  10. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Modelscited by 214
  11. 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 184
  12. 2020Enabling Language Models to Fill in the Blankscited by 121
6 more
  1. 2020Blank Language Modelscited by 43
  2. 2020Latin BERT: A Contextual Language Model for Classical Philologycited by 43
  3. 2019Text Infillingcited by 32
  4. 2017Bidirectional Beam Search: Forward-Backward Inference in Neural Sequence Models for Fill-in-the-Blank Image Captioningcited by 8
  5. 2023Deriving Language Models from Masked Language Modelscited by 8
  6. 2021What Do You Get When You Cross Beam Search with Nucleus Sampling?cited by 7

Built on by

Not cited yet.