# Adaptive Contrastive Search: Uncertainty-Guided Decoding for Open-Ended Text Generation

arXiv: https://arxiv.org/abs/2407.18698 · HTML: https://mlsanity.com/paper/2407.18698/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 36 references that are in the map.
- First posted on arXiv 26 Jul 2024.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges](https://mlsanity.com/paper/2011.06225/) (2020)
- [Contrastive Decoding: Open-ended Text Generation as Optimization](https://mlsanity.com/paper/2210.15097/) (2022)
- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://mlsanity.com/paper/2102.01454/) (2021)
- [A Deep Reinforced Model for Abstractive Summarization](https://mlsanity.com/paper/1705.04304/) (2017)
- [Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models](https://mlsanity.com/paper/1610.02424/) (2016)
- [OpenNMT: Open-source Toolkit for Neural Machine Translation](https://mlsanity.com/paper/1709.03815/) (2017)
- [A Contrastive Framework for Neural Text Generation](https://mlsanity.com/paper/2202.06417/) (2022)
- [Yara Parser: A Fast and Accurate Dependency Parser](https://mlsanity.com/paper/1503.06733/) (2015)
- [Contrastive Search Is What You Need For Neural Text Generation](https://mlsanity.com/paper/2210.14140/) (2022)
- [Consistency of a Recurrent Language Model With Respect to Incomplete Decoding](https://mlsanity.com/paper/2002.02492/) (2020)
- [Generating High-Quality and Informative Conversation Responses with Sequence-to-Sequence Models](https://mlsanity.com/paper/1701.03185/) (2017)
- [Fidelity-Enriched Contrastive Search: Reconciling the Faithfulness-Diversity Trade-Off in Text Generation](https://mlsanity.com/paper/2310.14981/) (2023)

## Built on by (most cited first, 27 of 27)

- [SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers](https://mlsanity.com/paper/2504.00255/) (2025)
- [Decoding Decoded: Understanding Hyperparameter Effects in Open-Ended Text Generation](https://mlsanity.com/paper/2410.06097/) (2024)
- [Enhancing Vision-Language Model Reliability with Uncertainty-Guided Dropout Decoding](https://mlsanity.com/paper/2412.06474/) (2024)
- [G2: Guided Generation for Enhanced Output Diversity in LLMs](https://mlsanity.com/paper/2511.00432/) (2025)
- [GUARD: Glocal Uncertainty-Aware Robust Decoding for Effective and Efficient Open-Ended Text Generation](https://mlsanity.com/paper/2508.20757/) (2025)
- [Towards Better Open-Ended Text Generation: A Multicriteria Evaluation Framework](https://mlsanity.com/paper/2410.18653/) (2024)
- [Statistical Multicriteria Evaluation of LLM-Generated Text](https://mlsanity.com/paper/2506.18082/) (2025)
- [Min-k Sampling: Decoupling Truncation from Temperature Scaling via Relative Logit Dynamics](https://mlsanity.com/paper/2604.11012/) (2026)
- [A Statistical Case Against Empirical Human-AI Alignment](https://mlsanity.com/paper/2502.14581/) (2025)
- [Avoidance Decoding for Diverse Multi-Branch Story Generation](https://mlsanity.com/paper/2509.02170/) (2025)
- [p-less Sampling: A Robust Hyperparameter-Free Approach for LLM Decoding](https://mlsanity.com/paper/2509.23234/) (2025)
- [Odysseus Navigates the Sirens' Song: Dynamic Focus Decoding for Factual and Diverse Open-Ended Text Generation](https://mlsanity.com/paper/2503.08057/) (2025)
- [The Geometry of Creative Variability: How Credal Sets Expose Calibration Gaps in Language Models](https://mlsanity.com/paper/2509.23088/) (2025)
- [LLMs Exhibit Significantly Lower Uncertainty in Creative Writing Than Professional Writers](https://mlsanity.com/paper/2602.16162/) (2026)
- [Multi-Amateur Contrastive Decoding for Text Generation](https://mlsanity.com/paper/2507.21086/) (2025)
- [Don't Throw Away Your Beams: Improving Consistency-based Uncertainties in LLMs via Beam Search](https://mlsanity.com/paper/2512.09538/) (2025)
- [The Truncation Blind Spot: How Decoding Strategies Systematically Exclude Human-Like Token Choices](https://mlsanity.com/paper/2603.18482/) (2026)
- [Pheromone-based Learning of Optimal Reasoning Paths](https://mlsanity.com/paper/2501.19278/) (2025)
- [Digitizing Nepal's Written Heritage: A Comprehensive HTR Pipeline for Old Nepali Manuscripts](https://mlsanity.com/paper/2512.17111/) (2025)
- [Confidence-Modulated Speculative Decoding for Large Language Models](https://mlsanity.com/paper/2508.15371/) (2025)
- [A Universal Avoidance Method for Diverse Multi-branch Generation](https://mlsanity.com/paper/2604.17323/) (2026)
- [Beyond Temperature: Hyperfitting as a Late-Stage Geometric Expansion](https://mlsanity.com/paper/2605.22579/) (2026)
- [Breaking the Likelihood Trap: Variance-Calibrated Modulation for Large Language Model Decoding](https://mlsanity.com/paper/2606.22511/) (2026)
- [Enhancing Learned Knowledge in LoRA Adapters Through Efficient Contrastive Decoding on Ascend NPUs](https://mlsanity.com/paper/2505.14620/) (2025)
- [Large Language Models Preserve Semantic Isotopies in Story Continuations](https://mlsanity.com/paper/2510.04400/) (2025)
- [Efficient Low-Resource Language Adaptation via Multi-Source Dynamic Logit Fusion](https://mlsanity.com/paper/2604.18106/) (2026)
- [Reference-Based Analysis of Coherence and Diversity in Open-Ended Text Generation](https://mlsanity.com/paper/2609.28080/) (2026)

## Abstract (arXiv)

Decoding from the output distributions of large language models to produce high-quality text is a complex challenge in language modeling. Various approaches, such as beam search, sampling with temperature, k-sampling, nucleus p-sampling, typical decoding, contrastive decoding, and contrastive search, have been proposed to address this problem, aiming to improve coherence, diversity, as well as resemblance to human-generated text. In this study, we introduce adaptive contrastive search, a novel decoding strategy extending contrastive search by incorporating an adaptive degeneration penalty, guided by the estimated uncertainty of the model at each generation step. This strategy is designed to enhance both the creativity and diversity of the language modeling process while at the same time producing coherent and high-quality generated text output. Our findings indicate performance enhancement in both aspects, across different model architectures and datasets, underscoring the effectiveness of our method in text generation tasks. Our code base, datasets, and models are publicly available.