# Decoding as Optimisation on the Probability Simplex: From Top-K to Top-P (Nucleus) to Best-of-K Samplers

arXiv: https://arxiv.org/abs/2602.18292 · HTML: https://mlsanity.com/paper/2602.18292/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 53 references that are in the map.
- First posted on arXiv 20 Feb 2026.

## Builds on (most cited first, 30 of 45)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024)
- [From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification](https://mlsanity.com/paper/1602.02068/) (2016)
- [Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models](https://mlsanity.com/paper/1610.02424/) (2016)
- [Universal Self-Consistency for Large Language Model Generation](https://mlsanity.com/paper/2311.17311/) (2023)
- [Controlled Decoding from Language Models](https://mlsanity.com/paper/2310.17022/) (2023)
- [If beam search is the answer, what was the question?](https://mlsanity.com/paper/2010.02650/) (2020)
- [Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs](https://mlsanity.com/paper/2305.11860/) (2023)
- [A Thorough Examination of Decoding Methods in the Era of LLMs](https://mlsanity.com/paper/2402.06925/) (2024)
- [Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Code](https://mlsanity.com/paper/2404.00971/) (2024)
- [Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity](https://mlsanity.com/paper/2510.01171/) (2025)
- [BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biology](https://mlsanity.com/paper/2503.00096/) (2025)
- [Reasoning with Sampling: Your Base Model is Smarter Than You Think](https://mlsanity.com/paper/2510.14901/) (2025)
- [MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation](https://mlsanity.com/paper/2410.11779/) (2024)
- [Tool Learning in the Wild: Empowering Language Models as Automatic Tool Agents](https://mlsanity.com/paper/2405.16533/) (2024)
- [Transfer Q Star: Principled Decoding for LLM Alignment](https://mlsanity.com/paper/2405.20495/) (2024)
- [Measuring and Reducing LLM Hallucination without Gold-Standard Answers](https://mlsanity.com/paper/2402.10412/) (2024)
- [Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening](https://mlsanity.com/paper/2601.21590/) (2026)
- [Accelerating Scientific Research with Gemini: Case Studies and Common Techniques](https://mlsanity.com/paper/2602.03837/) (2026)
- [Arithmetic Sampling: Parallel Diverse Decoding for Large Language Models](https://mlsanity.com/paper/2210.15458/) (2022)

## Built on by (most cited first, 2 of 2)

- [The Y-Combinator for LLMs: Solving Long-Context Rot with λ-Calculus](https://mlsanity.com/paper/2603.20105/) (2026)
- [Grounded Decoding: Retrieval-Anchored Probability Fusion for Faithful RAG](https://mlsanity.com/paper/2606.00432/) (2026)

## Abstract (arXiv)

Decoding sits between a language model and everything we do with it, yet it is still treated as a heuristic knob-tuning exercise. We argue decoding should be understood as a principled optimisation layer: at each token, we solve a regularised problem over the probability simplex that trades off model score against structural preferences and constraints. This single template recovers greedy decoding, Softmax sampling, Top-K, Top-P, and Sparsemax-style sparsity as special cases, and explains their common structure through optimality conditions. More importantly, the framework makes it easy to invent new decoders without folklore. We demonstrate this by designing Best-of-K (BoK), a KL-anchored coverage objective aimed at multi-sample pipelines (self-consistency, reranking, verifier selection). BoK targets the probability of covering good alternatives within a fixed K-sample budget and improves empirical performance. We show that such samples can improve accuracy by, for example, +18.6% for Qwen2.5-Math-7B on MATH500 at high sampling temperatures.