mlsanityData to 7 Oct 2026

A Thorough Examination of Decoding Methods in the Era of LLMs

arXiv 2402.06925 ↗First posted 10 Feb 2024cs.CLcited by 112

Not on the map. Why?

Abstract (arXiv)

Decoding methods play an indispensable role in converting language models from next-token predictors into practical task solvers. Prior research on decoding methods, primarily focusing on task-specific models, may not extend to the current era of general-purpose large language models (LLMs). Moreover, the recent influx of decoding strategies has further complicated this landscape. This paper provides a comprehensive and multifaceted analysis of various decoding methods within the context of LLMs, evaluating their performance, robustness to hyperparameter changes, and decoding speeds across a wide range of tasks, models, and deployment environments. Our findings reveal that decoding method performance is notably task-dependent and influenced by factors such as alignment, model size, and quantization. Intriguingly, sensitivity analysis exposes that certain methods achieve superior performance at the cost of extensive hyperparameter tuning, highlighting the trade-off between attaining optimal results and the practicality of implementation in varying contexts.

Key facts. Cited by 112 papers in the map (data through 7 Oct 2026). Builds on 31 of its 61 references that are in the map. First posted on arXiv 10 Feb 2024.

Builds on

31 of 61 references.

References · 31

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  8. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  9. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  10. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  11. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  12. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
19 more
  1. 2018Hierarchical Neural Story Generationcited by 1,182
  2. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  3. 2015Teaching Machines to Read and Comprehendcited by 1,039
  4. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  5. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  6. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  7. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  8. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  9. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  10. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  11. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
  12. 2017Beam Search Strategies for Neural Machine Translationcited by 232
  13. 2022A Contrastive Framework for Neural Text Generationcited by 224
  14. 2022Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionalscited by 191
  15. 2022Truncation Sampling as Language Model Desmoothingcited by 121
  16. 2022On Decoding Strategies for Neural Text Generatorscited by 89
  17. 2019Comparison of Diverse Decoding Methods from Conditional Language Modelscited by 78
  18. 2023Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspectivecited by 39
  19. 2023A Frustratingly Simple Decoding Method for Neural Text Generationcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 25; 2025: 42; 2026: 45.

112 papers.

Sort:

By area:

  1. 2024ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generationcited by 118
  2. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  3. 2024InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructionscited by 54
  4. 2024BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Modelscited by 44
  5. 2024Robustifying Safety-Aligned Large Language Models through Clean Data Curationcited by 43
  6. 2024Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examplescited by 29
  7. 2024Decoding Decoded: Understanding Hyperparameter Effects in Open-Ended Text Generationcited by 26
  8. 2025LLM Probability Concentration: How Alignment Shrinks the Generative Horizoncited by 26
  9. 2024HoLLMwood: Unleashing the Creativity of Large Language Models in Screenwriting via Role Playingcited by 22
  10. 2024Practical and efficient quantum circuit synthesis and transpiling with Reinforcement Learningcited by 21
  11. 2024Unchosen Experts Can Contribute Too: Unleashing MoE Models' Power by Self-Contrastcited by 19
  12. 2024Balancing Diversity and Risk in LLM Sampling: How to Select Your Method and Parameter for Open-Ended Text Generationcited by 18
38 more
  1. 2025HermesFlow: Seamlessly Closing the Gap in Multimodal Understanding and Generationcited by 17
  2. 2024SLED: Self Logits Evolution Decoding for Improving Factuality in Large Language Modelscited by 16
  3. 2025Control the Temperature: Selective Sampling for Diverse and High-Quality LLM Outputscited by 15
  4. 2025Do Repetitions Matter? Strengthening Reliability in LLM Evaluationscited by 14
  5. 2025Provable Post-Training Quantization: Theoretical Analysis of OPTQ and Qronoscited by 13
  6. 2024Looking Beyond The Top-1: Transformers Determine Top Tokens In Ordercited by 12
  7. 2025Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metricscited by 12
  8. 2026Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examinationcited by 12
  9. 2024Optimized Multi-Token Joint Decoding with Auxiliary Model for LLM Inferencecited by 11
  10. 2024LLM2: Let Large Language Models Harness System 2 Reasoningcited by 11
  11. 2024Retrieval is Accurate Generationcited by 10
  12. 2025Mitigating Hallucination in Large Vision-Language Models via Adaptive Attention Calibrationcited by 10
  13. 2025Active Layer-Contrastive Decoding Reduces Hallucination in Large Language Model Generationcited by 10
  14. 2025GUARD: Glocal Uncertainty-Aware Robust Decoding for Effective and Efficient Open-Ended Text Generationcited by 10
  15. 2024Collaborative decoding of critical tokens for boosting factuality of large language modelscited by 9
  16. 2024Adaptive Decoding via Latent Preference Optimizationcited by 9
  17. 2025TTPA: Token-level Tool-use Preference Alignment Training Framework with Fine-grained Evaluationcited by 9
  18. 2025When Life Gives You Samples: The Benefits of Scaling up Inference Compute for Multilingual LLMscited by 9
  19. 2025Privacy-Aware Decoding: Mitigating Privacy Leakage of Large Language Models in Retrieval-Augmented Generationcited by 8
  20. 2026From Narrow to Panoramic Vision: Attention-Guided Cold-Start Reshapes Multimodal Reasoningcited by 8
  21. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decodingcited by 7
  22. 2024Decoding Game: On Minimax Optimality of Heuristic Text Generation Strategiescited by 6
  23. 2025Hallucinate at the Last in Long Response Generation: A Case Study on Long Document Summarizationcited by 6
  24. 2026Rare Event Analysis of Large Language Modelscited by 6
  25. 2026UReason: Benchmarking Reasoning-to-Generation Alignment in Unified Multimodal Modelscited by 6
  26. 2026The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibilitycited by 6
  27. 2024Dynamic-Width Speculative Beam Decoding for Efficient LLM Inferencecited by 5
  28. 2025Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMscited by 5
  29. 2025The End of Manual Decoding: Towards Truly End-to-End Language Modelscited by 5
  30. 2025Think in Parallel, Answer as One: Logit Averaging for Open-Ended Reasoningcited by 5
  31. 2026Learning to Explore with Parameter-Space Noise: A Deep Dive into Parameter-Space Noise for Reinforcement Learning with Verifiable Rewardscited by 5
  32. 2026On Calibration of Large Language Models: From Response To Capabilitycited by 5
  33. 2024Impact of Decoding Methods on Human Alignment of Conversational LLMscited by 4
  34. 2025Advancing Decoding Strategies: Enhancements in Locally Typical Sampling for LLMscited by 4
  35. 2025Toward Principled LLM Safety Testing: Solving the Jailbreak Oracle Problemcited by 4
  36. 2025Bridging On-Device and Cloud LLMs for Collaborative Reasoning: A Unified Methodology for Local Routing and Post-Trainingcited by 4
  37. 2025Decoding Uncertainty: The Impact of Decoding Strategies for Uncertainty Estimation in Large Language Modelscited by 3
  38. 2025Generalization or Memorization: Dynamic Decoding for Mode Steeringcited by 3

50 most cited of 112 All 112 (JSON)