# Diversity Explains Inference Scaling Laws: Through a Case Study of Minimum Bayes Risk Decoding

arXiv: https://arxiv.org/abs/2410.15021 · HTML: https://mlsanity.com/paper/2410.15021/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 78 references that are in the map.
- First posted on arXiv 19 Oct 2024.

## Builds on (most cited first, 30 of 37)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://mlsanity.com/paper/2407.21787/) (2024)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization](https://mlsanity.com/paper/1911.12237/) (2019)
- [nocaps: novel object captioning at scale](https://mlsanity.com/paper/1812.08658/) (2018)
- [Complexity-Based Prompting for Multi-Step Reasoning](https://mlsanity.com/paper/2210.00720/) (2022)
- [Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models](https://mlsanity.com/paper/2408.00724/) (2024)
- [Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation](https://mlsanity.com/paper/2005.10283/) (2020)
- [Truncation Sampling as Language Model Desmoothing](https://mlsanity.com/paper/2210.15191/) (2022)
- [High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics](https://mlsanity.com/paper/2111.09388/) (2021)
- [A Unified Theory of Diversity in Ensemble Learning](https://mlsanity.com/paper/2301.03962/) (2023)
- [Sampling-Based Approximations to Minimum Bayes Risk Decoding for Neural Machine Translation](https://mlsanity.com/paper/2108.04718/) (2021)
- [Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translation](https://mlsanity.com/paper/2305.09860/) (2023)
- [Understanding the Properties of Minimum Bayes Risk Decoding in Neural Machine Translation](https://mlsanity.com/paper/2105.08504/) (2021)
- [Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decoding](https://mlsanity.com/paper/2211.07634/) (2022)
- [It's MBR All the Way Down: Modern Generation Techniques Through the Lens of Minimum Bayes Risk](https://mlsanity.com/paper/2310.01387/) (2023)
- [Identifying Weaknesses in Machine Translation Metrics Through Minimum Bayes Risk Decoding: A Case Study for COMET](https://mlsanity.com/paper/2202.05148/) (2022)
- [Faster Minimum Bayes Risk Decoding with Confidence-based Pruning](https://mlsanity.com/paper/2311.14919/) (2023)
- [Linear-time Minimum Bayes Risk Decoding with Reference Aggregation](https://mlsanity.com/paper/2402.04251/) (2024)
- [Neural Machine Translation by Minimising the Bayes-risk with Respect to Syntactic Translation Lattices](https://mlsanity.com/paper/1612.03791/) (2016)
- [Less Is More: A Comprehensive Framework for the Number of Components of Ensemble Classifiers](https://mlsanity.com/paper/1709.02925/) (2017)

## Built on by (most cited first, 9 of 9)

- [Where does output diversity collapse in post-training?](https://mlsanity.com/paper/2604.16027/) (2026)
- [Case-Based Decision-Theoretic Decoding with Quality Memories](https://mlsanity.com/paper/2509.12677/) (2025)
- [Minimum Bayes Risk Decoding for Error Span Detection in Reference-Free Automatic Machine Translation Evaluation](https://mlsanity.com/paper/2512.07540/) (2025)
- [Is Human Annotation Necessary? Iterative MBR Distillation for Error Span Detection in Machine Translation](https://mlsanity.com/paper/2603.12983/) (2026)
- [Enhancing Factuality through Consensus and Consistency in Summarization Using Minimum Bayes Risk Decoding](https://mlsanity.com/paper/2605.29336/) (2026)
- [Non-Autoregressive Minimum Bayes' Risk Decoding for Fast Speech Recognition](https://mlsanity.com/paper/2606.17537/) (2026)
- [Quit While You're Ahead: Quit for Efficient Candidate Generation in Machine Translation Reranking](https://mlsanity.com/paper/2609.00588/) (2026)
- [Overfitting Mitigation via Singular Value Decomposition in Minimum Bayes Risk Decoding](https://mlsanity.com/paper/2609.01135/) (2026)
- [From Sampled Evidence to Radiology Reports: Objectives, Constraints, and Realization](https://mlsanity.com/paper/2610.17318/) (2026)

## Abstract (arXiv)

Inference methods play an important role in eliciting the performance of large language models (LLMs). Currently, LLMs use inference methods utilizing generated multiple samples, which can be derived from Minimum Bayes Risk (MBR) Decoding. Previous studies have conducted empirical analyses to clarify the improvements in generation performance achieved by MBR decoding and have reported various observations. However, the theoretical underpinnings of these findings remain uncertain. To address this, we offer a new theoretical interpretation of MBR decoding from the perspective of bias-diversity decomposition. In this interpretation, the error in the quality estimation of hypotheses by MBR decoding is decomposed into two main factors: bias, which considers the closeness between the utility function and human evaluation, and diversity, which represents the variability in the quality estimation of the utility function. The theoretical analysis reveals the difficulty of simultaneously improving bias and diversity, confirming the validity of enhancing MBR decoding performance by increasing diversity. Furthermore, we reveal that diversity can explain one aspect of inference scaling laws that describe performance improvement by increasing sample size. Moreover, experiments across multiple NLP tasks yielded results consistent with these theoretical characteristics. Our code is available at https://github.com/naist-nlp/mbr-bias-diversity.