mlsanityData to 7 Oct 2026

Generating Diverse and High-Quality Texts by Minimum Bayes Risk Decoding

arXiv 2401.05054 ↗First posted 10 Jan 2024cs.CL cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

One of the most important challenges in text generation systems is to produce outputs that are not only correct but also diverse. Recently, Minimum Bayes-Risk (MBR) decoding has gained prominence for generating sentences of the highest quality among the decoding algorithms. However, existing algorithms proposed for generating diverse outputs are predominantly based on beam search or random sampling, thus their output quality is capped by these underlying methods. In this paper, we investigate an alternative approach -- we develop diversity-promoting decoding algorithms by enforcing diversity objectives to MBR decoding. We propose two variants of MBR, Diverse MBR (DMBR) and k-medoids MBR (KMBR), methods to generate a set of sentences with high quality and diversity. We evaluate DMBR and KMBR on a variety of directed text generation tasks using encoder-decoder models and a large language model with prompting. The experimental results show that the proposed method achieves a better trade-off than the diverse beam search and sampling algorithms.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 48 of its 80 references that are in the map. First posted on arXiv 10 Jan 2024.

Builds on

48 of 80 references.

References · 48

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  10. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  11. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  12. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
36 more
  1. 2018Hierarchical Neural Story Generationcited by 1,182
  2. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  3. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  4. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  5. 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 299
  6. 2019Stochastic Beams and Where to Find Them: The Gumbel-Top-k Trick for Sampling Sequences Without Replacementcited by 205
  7. 2019Facebook FAIR's WMT19 News Translation Task Submissioncited by 179
  8. 2020Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translationcited by 137
  9. 2020Evaluating the Evaluation of Diversity in Natural Language Generationcited by 127
  10. 2022Truncation Sampling as Language Model Desmoothingcited by 121
  11. 2019On NMT Search Errors and Model Errors: Cat Got Your Tongue?cited by 114
  12. 2020If beam search is the answer, what was the question?cited by 114
  13. 2016A Hierarchical Approach for Generating Descriptive Image Paragraphscited by 104
  14. 2021High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metricscited by 104
  15. 2020Trading Off Diversity and Quality in Natural Language Generationcited by 94
  16. 2019Comparison of Diverse Decoding Methods from Conditional Language Modelscited by 78
  17. 2019Mixture Models for Diverse Machine Translation: Tricks of the Tradecited by 77
  18. 2021Sampling-Based Approximations to Minimum Bayes Risk Decoding for Neural Machine Translationcited by 63
  19. 2023Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translationcited by 57
  20. 2021Understanding the Properties of Minimum Bayes Risk Decoding in Neural Machine Translationcited by 56
  21. 2022Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decodingcited by 54
  22. 2019Describing like humans: on diversity in image captioningcited by 46
  23. 2018Importance of Search and Evaluation Strategies in Neural Dialogue Modelingcited by 44
  24. 2016Mutual Information and Diverse Decoding Improve Neural Machine Translationcited by 35
  25. 2023Faster Minimum Bayes Risk Decoding with Confidence-based Pruningcited by 34
  26. 2019Complexity-Weighted Loss and Diverse Reranking for Sentence Simplificationcited by 20
  27. 2016Noisy Parallel Approximate Decoding for Conditional Recurrent Language Modelcited by 18
  28. 2021Determinantal Beam Searchcited by 18
  29. 2024Hyperparameter-Free Approach for Faster Minimum Bayes Risk Decodingcited by 15
  30. 2021What's The Latest? A Question-driven News Chatbotcited by 14
  31. 2017Greed is Good: Near-Optimal Submodular Maximization via Greedy Optimizationcited by 11
  32. 2023Look-back Decoding for Open-Ended Text Generationcited by 10
  33. 2019Generating Diverse Story Continuations with Controllable Semanticscited by 9
  34. 2022Best-k Search Algorithm for Neural Text Generationcited by 6
  35. 2019Guarantees of Stochastic Greedy Algorithms for Non-monotone Submodular Maximization with Cardinality Constraintcited by 4
  36. 2017Domain Aware Neural Dialog Systemcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 3; 2026: 3.

11 papers.

  1. 2024QUEST: Quality-Aware Metropolis-Hastings Sampling for Machine Translationcited by 16
  2. 2025Control the Temperature: Selective Sampling for Diverse and High-Quality LLM Outputscited by 15
  3. 2024mbrs: A Library for Minimum Bayes Risk Decodingcited by 13
  4. 2025Revisiting Compositional Generalization Capability of Large Language Models Considering Instruction Following Abilitycited by 10
  5. 2024Diversity Explains Inference Scaling Laws: Through a Case Study of Minimum Bayes Risk Decodingcited by 8
  6. 2024Annotation-Efficient Language Model Alignment via Diverse and Representative Response Textscited by 3
  7. 2024On the True Distribution Approximation of Minimum Bayes-Risk Decodingcited by 2
  8. 2025Exploring the Relationship Between Diversity and Quality in Ad Text Generationcited by 0
  9. 2026Large Language Models as Amortized Pareto-Front Generators for Constrained Bi-Objective Convex Optimizationcited by 0
  10. 2026How Neural Reward Models Learn Features for Policy Optimization: A Single-Index Analysiscited by 0
  11. 2026TalkMatrix: Generating Character Dialogue that is Both Consistent and Diversecited by 0