# Balancing Diversity and Risk in LLM Sampling: How to Select Your Method and Parameter for Open-Ended Text Generation

arXiv: https://arxiv.org/abs/2408.13586 · HTML: https://mlsanity.com/paper/2408.13586/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 37 references that are in the map.
- First posted on arXiv 24 Aug 2024.

## Builds on (most cited first, 19 of 19)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [Neural Text Generation with Unlikelihood Training](https://mlsanity.com/paper/1908.04319/) (2019)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://mlsanity.com/paper/2102.01454/) (2021)
- [Chain-of-Thought Reasoning Without Prompting](https://mlsanity.com/paper/2402.10200/) (2024)
- [Truncation Sampling as Language Model Desmoothing](https://mlsanity.com/paper/2210.15191/) (2022)
- [A Thorough Examination of Decoding Methods in the Era of LLMs](https://mlsanity.com/paper/2402.06925/) (2024)
- [Improving Open-Ended Text Generation via Adaptive Decoding](https://mlsanity.com/paper/2402.18223/) (2024)
- [Pointwise Mutual Information Based Metric and Decoding Strategy for Faithful Generation in Document Grounded Dialogs](https://mlsanity.com/paper/2305.12191/) (2023)

## Built on by (most cited first, 19 of 19)

- [Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models](https://mlsanity.com/paper/2412.02980/) (2024)
- [Navigate the Unknown: Enhancing LLM Reasoning with Intrinsic Motivation Guided Exploration](https://mlsanity.com/paper/2505.17621/) (2025)
- [Decoding Decoded: Understanding Hyperparameter Effects in Open-Ended Text Generation](https://mlsanity.com/paper/2410.06097/) (2024)
- [On the Effect of Sampling Diversity in Scaling LLM Inference](https://mlsanity.com/paper/2502.11027/) (2025)
- [LongEval: A Comprehensive Analysis of Long-Text Generation Through a Plan-based Paradigm](https://mlsanity.com/paper/2502.19103/) (2025)
- [Min-k Sampling: Decoupling Truncation from Temperature Scaling via Relative Logit Dynamics](https://mlsanity.com/paper/2604.11012/) (2026)
- [p-less Sampling: A Robust Hyperparameter-Free Approach for LLM Decoding](https://mlsanity.com/paper/2509.23234/) (2025)
- [Ontologies in Design: How Imagining a Tree Reveals Possibilites and Assumptions in Large Language Models](https://mlsanity.com/paper/2504.03029/) (2025)
- [Principled Coarse-Grained Acceptance for Speculative Decoding in Speech](https://mlsanity.com/paper/2511.13732/) (2025)
- [Decoding as Optimisation on the Probability Simplex: From Top-K to Top-P (Nucleus) to Best-of-K Samplers](https://mlsanity.com/paper/2602.18292/) (2026)
- [Sampling More, Getting Less: Calibration is the Diversity Bottleneck in LLMs](https://mlsanity.com/paper/2605.11128/) (2026)
- [Short-Context Dominance: How Much Local Context Natural Language Actually Needs?](https://mlsanity.com/paper/2512.08082/) (2025)
- [Quality-constrained Entropy Maximization Policy Optimization for LLM Diversity](https://mlsanity.com/paper/2602.15894/) (2026)
- [Breaking the Likelihood Trap: Variance-Calibrated Modulation for Large Language Model Decoding](https://mlsanity.com/paper/2606.22511/) (2026)
- [ARB: A Matched Authorship-Rewriting Benchmark Dataset for AI-Text Detector Evaluation](https://mlsanity.com/paper/2607.29539/) (2026)
- [Revisiting Lossy Verification in Speculative Decoding: Mechanisms, Trade-offs, and Failure Modes](https://mlsanity.com/paper/2607.26627/) (2026)
- [Human-Simulation Interaction: From Prediction to Exploration in LLM Agent Simulations for Policy](https://mlsanity.com/paper/2608.07496/) (2026)
- [Generate to Explore, Select to Exploit: Aligning LLM-based Headline Generation with Personalized Recommendation](https://mlsanity.com/paper/2609.15094/) (2026)
- [Truncation Samplers Are Nested: Comparing Decoding Rules Is a Measurement Problem](https://mlsanity.com/paper/2610.15570/) (2026)

## Abstract (arXiv)

Sampling-based decoding strategies have been widely adopted for Large Language Models (LLMs) in numerous applications, targeting a balance between diversity and quality via temperature tuning and tail truncation. Considering the strong dependency of the candidate next tokens on different prefixes, recent studies propose to adaptively truncate the tail of LLMs' predicted distribution. Although improved results have been reported with these methods on open-ended text generation tasks, the results are highly dependent on the curated parameters and the limited exemplar text. In this paper, we propose a systematic way to estimate the capacity of a truncation sampling method by considering the trade-off between diversity and risk at each decoding step, based on our collected prefix tree which preserves the context of a full sentence. Our work offers a comprehensive comparison of existing truncation sampling methods and serves as a practical user guideline for their parameter selection.