mlsanityData to 7 Oct 2026

BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model

arXiv 1902.04094 ↗First posted 11 Feb 2019cs.CL cs.LGcited by 184

Not on the map. Why?

Cited by 26 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We show that BERT (Devlin et al., 2018) is a Markov random field language model. This formulation gives way to a natural procedure to sample sentences from BERT. We generate from BERT and find that it can produce high-quality, fluent generations. Compared to the generations of a traditional left-to-right language model, BERT generates sentences that are more diverse but of slightly worse quality.

Key facts. Cited by 184 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 39; 2022: 41; 2023: 34; 2024: 31; 2025: 23; 2026: 16.

184 papers.

Sort:

By area:

  1. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  2. 2024Simple and Effective Masked Diffusion Language Modelscited by 800
  3. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  4. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 712
  5. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  6. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  7. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
  8. 2021Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Modelscited by 251
  9. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Modelscited by 214
  10. 2022Continuous diffusion for categorical datacited by 202
  11. 2021Societal Biases in Language Generation: Progress and Challengescited by 201
  12. 2022"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Datasetcited by 160
38 more
  1. 2024Simple Guidance Mechanisms for Discrete Diffusion Modelscited by 148
  2. 2025The Diffusion Dualitycited by 124
  3. 2021Step-unrolled Denoising Autoencoders for Text Generationcited by 102
  4. 2022What Artificial Neural Networks Can Tell Us About Human Language Acquisitioncited by 97
  5. 2021An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Modelscited by 96
  6. 2023Amortizing intractable inference in large language modelscited by 94
  7. 2021Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?cited by 91
  8. 2025Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Reviewcited by 82
  9. 2022On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representationscited by 75
  10. 2021Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codescited by 71
  11. 2024Attention Heads of Large Language Models: A Surveycited by 65
  12. 2022A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyondcited by 64
  13. 2023Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpuscited by 62
  14. 2021Unmasking the Mask -- Evaluating Social Biases in Masked Language Modelscited by 60
  15. 2023Right to be Forgotten in the Era of Large Language Models: Implications, Challenges, and Solutionscited by 59
  16. 2021BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?cited by 56
  17. 2022POLITICS: Pretraining with Same-story Article Comparison for Ideology Prediction and Stance Detectioncited by 50
  18. 2022Bidirectional Language Models Are Also Few-shot Learnerscited by 48
  19. 2022Debiasing isn't enough! -- On the Effectiveness of Debiasing MLMs and their Social Biases in Downstream Taskscited by 48
  20. 2021CLiMP: A Benchmark for Chinese Language Model Evaluationcited by 44
  21. 2022Event knowledge in large language models: the gap between the impossible and the unlikelycited by 44
  22. 2022A Langevin-like Sampler for Discrete Distributionscited by 42
  23. 2021Tracing Text Provenance via Context-Aware Lexical Substitutioncited by 40
  24. 2023Trained on 100 million words and still in shape: BERT meets British National Corpuscited by 39
  25. 2021Exposing the Implicit Energy Networks behind Masked Language Models via Metropolis--Hastingscited by 37
  26. 2022Show Me What and Tell Me How: Video Synthesis via Multimodal Conditioningcited by 37
  27. 2023Neural Retrievers are Biased Towards LLM-Generated Contentcited by 37
  28. 2021M6-UFC: Unifying Multi-Modal Controls for Conditional Image Synthesis via Non-Autoregressive Generative Transformerscited by 36
  29. 2022SPACE-3: Unified Dialog Model Pre-training for Task-Oriented Dialog Understanding and Generationcited by 34
  30. 2021BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamesecited by 33
  31. 2022Fewer Errors, but More Stereotypes? The Effect of Model Size on Gender Biascited by 33
  32. 2023ConZIC: Controllable Zero-shot Image Captioning by Sampling-Based Polishingcited by 33
  33. 2021Do language models learn typicality judgments from text?cited by 32
  34. 2022Using Pre-Trained Language Models for Producing Counter Narratives Against Hate Speech: a Comparative Studycited by 30
  35. 2022Draft-and-Revise: Effective Image Generation with Contextual RQ-Transformercited by 30
  36. 2025Continuous Diffusion Model for Language Modelingcited by 29
  37. 2023A Better Way to Do Masked Language Model Scoringcited by 27
  38. 2024GPT or BERT: why not both?cited by 26

50 most cited of 184 All 184 (JSON)