# On the Effect of Sampling Diversity in Scaling LLM Inference

arXiv: https://arxiv.org/abs/2502.11027 · HTML: https://mlsanity.com/paper/2502.11027/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 71 of its 87 references that are in the map.
- First posted on arXiv 16 Feb 2025.

## Builds on (most cited first, 30 of 71)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024)
- [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://mlsanity.com/paper/2407.21787/) (2024)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [Reasoning with Language Model is Planning with World Model](https://mlsanity.com/paper/2305.14992/) (2023)
- [Automatic Chain of Thought Prompting in Large Language Models](https://mlsanity.com/paper/2210.03493/) (2022)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [PAL: Program-aided Language Models](https://mlsanity.com/paper/2211.10435/) (2022)

## Built on by (most cited first, 18 of 18)

- [Mode-Conditioning Unlocks Superior Test-Time Scaling](https://mlsanity.com/paper/2512.01127/) (2025)
- [Uncertainty-Aware Answer Selection for Improved Reasoning in Multi-LLM Systems](https://mlsanity.com/paper/2510.02377/) (2025)
- [PathWise: Planning through World Model for Automated Heuristic Design via Self-Evolving LLMs](https://mlsanity.com/paper/2601.20539/) (2026)
- [When More Sampling Hurts: The Modal Ceiling and Correlation Ceiling of Test-Time Scaling](https://mlsanity.com/paper/2606.28661/) (2026)
- [Generalizing Test-time Compute-optimal Scaling as an Optimizable Graph](https://mlsanity.com/paper/2511.00086/) (2025)
- [Confidence-Weighted Token Set Cover for Early Hypothesis Pruning in Self-Consistency](https://mlsanity.com/paper/2508.03979/) (2025)
- [T-pro 2.0: An Efficient Russian Hybrid-Reasoning Model and Playground](https://mlsanity.com/paper/2512.10430/) (2025)
- [Decoding as Optimisation on the Probability Simplex: From Top-K to Top-P (Nucleus) to Best-of-K Samplers](https://mlsanity.com/paper/2602.18292/) (2026)
- [Input-Time Scaling: Adding Noise and Irrelevance into Less-Is-More Drastically Improves Reasoning Performance and Efficiency](https://mlsanity.com/paper/2508.13654/) (2025)
- [A Formal Framework for Uncertainty Analysis of Text Generation with Large Language Models](https://mlsanity.com/paper/2603.26363/) (2026)
- [SonicSampler: Unified Tile-Aware Kernels for LLM Sampling and Speculative Verification](https://mlsanity.com/paper/2607.20475/) (2026)
- [Planned Test-Time Scaling with Coordinated Reasoning Paths](https://mlsanity.com/paper/2609.27374/) (2026)
- [Strategically Diverse Sampling for Self-Training](https://mlsanity.com/paper/2609.31571/) (2026)
- [Don't Repeat Yourself: Self-Supervised Fine-Tuning for Coverage](https://mlsanity.com/paper/2609.31688/) (2026)
- [Improving the Diversity of LLM Outputs without a Trade-off](https://mlsanity.com/paper/2609.33038/) (2026)
- [Shockingly Simple Self-retrospection Improves Agentic Models Without RL](https://mlsanity.com/paper/2609.35741/) (2026)
- [Explore Broadly, Reason Sharply: Push Small Models toward the Frontier via Sampling](https://mlsanity.com/paper/2609.38104/) (2026)
- [From Discovery to Decision: Finite-Budget Recoverability in LLM Voting](https://mlsanity.com/paper/2610.01014/) (2026)

## Abstract (arXiv)

Large language model (LLM) scaling inference is key to unlocking greater performance, and leveraging diversity has proven an effective way to enhance it. Motivated by the observed relationship between solution accuracy and meaningful response diversity, we systematically study the effect of prompt diversity in scaling inference. We theoretically explain why diversified sampling improves Best-of-N scaling, showing that responses generated from diverse prompts after Best-of-N selection exhibit significantly lower error rates than those produced from stationary prompts. Building on this analysis, we derive a diversity-fidelity trade-off principle, that guides the design of sampling strategies introducing diversity. From this guidance, we instantiate a family of effective perturbation styles. We theoretically and empirically characterize when diversified exploration remains effective, demonstrating that it works under a variety of conditions, and we further show that under majority voting, diversity may vanish. Finally, we systematically evaluate the effectiveness of sampling diversity and show that, when applied appropriately in different contexts, meaningful perturbations yield stronger, task-dependent gains as diversity increases. Overall, this work provides a systematic analysis that offers a theoretical and empirical foundation for understanding how sampling diversity affects LLM inference-time scaling.