mlsanityData to 7 Oct 2026

Decoding as Optimisation on the Probability Simplex: From Top-K to Top-P (Nucleus) to Best-of-K Samplers

arXiv 2602.18292 ↗First posted 20 Feb 2026cs.LG cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Decoding sits between a language model and everything we do with it, yet it is still treated as a heuristic knob-tuning exercise. We argue decoding should be understood as a principled optimisation layer: at each token, we solve a regularised problem over the probability simplex that trades off model score against structural preferences and constraints. This single template recovers greedy decoding, Softmax sampling, Top-K, Top-P, and Sparsemax-style sparsity as special cases, and explains their common structure through optimality conditions. More importantly, the framework makes it easy to invent new decoders without folklore. We demonstrate this by designing Best-of-K (BoK), a KL-anchored coverage objective aimed at multi-sample pipelines (self-consistency, reranking, verifier selection). BoK targets the probability of covering good alternatives within a fixed K-sample budget and improves empirical performance. We show that such samples can improve accuracy by, for example, +18.6% for Qwen2.5-Math-7B on MATH500 at high sampling temperatures.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 45 of its 53 references that are in the map. First posted on arXiv 20 Feb 2026.

Builds on

45 of 53 references.

References · 45

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2017On Calibration of Modern Neural Networkscited by 4,860
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  6. 2023Let's Verify Step by Stepcited by 3,598
  7. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  9. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  10. 2021Convex optimizationcited by 2,479
  11. 2018Hierarchical Neural Story Generationcited by 1,182
  12. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
33 more
  1. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 337
  2. 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 299
  3. 2023Universal Self-Consistency for Large Language Model Generationcited by 198
  4. 2023Controlled Decoding from Language Modelscited by 133
  5. 2020If beam search is the answer, what was the question?cited by 114
  6. 2024A Thorough Examination of Decoding Methods in the Era of LLMscited by 112
  7. 2023Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMscited by 111
  8. 2024Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Codecited by 106
  9. 2025Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversitycited by 100
  10. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigationcited by 87
  11. 2025Reasoning with Sampling: Your Base Model is Smarter Than You Thinkcited by 84
  12. 2025BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biologycited by 75
  13. 2024Tool Learning in the Wild: Empowering Language Models as Automatic Tool Agentscited by 56
  14. 2024Transfer Q Star: Principled Decoding for LLM Alignmentcited by 46
  15. 2024Measuring and Reducing LLM Hallucination without Gold-Standard Answerscited by 31
  16. 2026Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpeningcited by 27
  17. 2026Accelerating Scientific Research with Gemini: Case Studies and Common Techniquescited by 24
  18. 2022Arithmetic Sampling: Parallel Diverse Decoding for Large Language Modelscited by 18
  19. 2024Balancing Diversity and Risk in LLM Sampling: How to Select Your Method and Parameter for Open-Ended Text Generationcited by 18
  20. 2025On the Effect of Sampling Diversity in Scaling LLM Inferencecited by 18
  21. 2023Why Can Large Language Models Generate Correct Chain-of-Thoughts?cited by 16
  22. 2025A Systematic Literature Review of Retrieval-Augmented Generation: Techniques, Metrics, and Challengescited by 16
  23. 2026LLM-42: Enabling Determinism in LLM Inference with Verified Speculationcited by 14
  24. 2024Creative Beam Search: LLM-as-a-Judge For Improving Response Generationcited by 13
  25. 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Timecited by 13
  26. 2025Foundations of Top-k Decoding For Language Modelscited by 10
  27. 2025Bourbaki: Self-Generated and Goal-Conditioned MDPs for Theorem Provingcited by 10
  28. 2024PhysicsAssistant: An LLM-Powered Interactive Learning Robot for Physics Lab Investigationscited by 8
  29. 2025Scaled-Dot-Product Attention as One-Sided Entropic Optimal Transportcited by 7
  30. 2025Model-Based and Sample-Efficient AI-Assisted Math Discovery in Sphere Packingcited by 2
  31. 2025BEAVER: An Efficient Deterministic LLM Verifiercited by 2
  32. 2026Task-Awareness Improves LLM Generations and Uncertaintycited by 2
  33. 2025Rethinking Large Language Model Distillation: A Constrained Markov Decision Process Perspectivecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026The Y-Combinator for LLMs: Solving Long-Context Rot with λ-Calculuscited by 3
  2. 2026Grounded Decoding: Retrieval-Anchored Probability Fusion for Faithful RAGcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026XTC: Head-Aware Sampling by Excluding Top Choices
  2. 2025p-less Sampling: A Robust Hyperparameter-Free Approach for LLM Decoding
  3. 2025From Bits to Rounds: Parallel Decoding with Exploration for Diffusion Language Models
  4. 2026Sampling for Quality: Training-Free Reward-Guided LLM Decoding via Sequential Monte Carlo
  5. 2026Entropy-informed Decoding: Adaptive Information-Driven Branching
  6. 2026Adaptive Decoding via Test-Time Policy Learning for Self-Improving Generation
  7. 2025Sampling-Efficient Test-Time Scaling: Self-Estimating the Best-of-N Sampling in Early Decoding
  8. 2026Locally Confident, Globally Stuck: The Quality-Exploration Dilemma in Diffusion Language Models