mlsanityData to 7 Oct 2026

Contrastive Decoding: Open-ended Text Generation as Optimization

arXiv 2210.15097 ↗First posted 27 Oct 2022cs.CL cs.AI cs.LGcited by 598

Not on the map. Why?

Cited by 106 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Given a language model (LM), maximum probability is a poor decoding objective for open-ended generation, because it produces short and repetitive text. On the other hand, sampling can often produce incoherent text that drifts from the original topics. We propose contrastive decoding (CD), a reliable decoding approach that optimizes a contrastive objective subject to a plausibility constraint. The contrastive objective returns the difference between the likelihood under a large LM (called the expert, e.g. OPT-13B) and a small LM (called the amateur, e.g. OPT-125M), and the constraint ensures that the outputs are plausible. CD is inspired by the fact that the failures of larger LMs (e.g., repetition, incoherence) are even more prevalent in smaller LMs, and that this difference signals which texts should be preferred. CD requires zero additional training, and produces higher quality text than decoding from the larger LM alone. It also works across model scales (OPT-13B and GPT2-1.5B) and significantly outperforms four strong decoding algorithms (e.g., nucleus, top-k) in automatic and human evaluations across wikipedia, news and story domains.

Key facts. Cited by 598 papers in the map (data through 7 Oct 2026). Builds on 23 of its 39 references that are in the map. First posted on arXiv 27 Oct 2022.

Builds on

23 of 39 references.

References · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  3. 2016Pointer Sentinel Mixture Modelscited by 2,992
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  6. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  7. 2018Hierarchical Neural Story Generationcited by 1,182
  8. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  9. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  10. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  11. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  12. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
11 more
  1. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  2. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  3. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  4. 2022A Contrastive Framework for Neural Text Generationcited by 224
  5. 2016Professor Forcing: A New Algorithm for Training Recurrent Networkscited by 214
  6. 2020Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translationcited by 137
  7. 2016Sequence-to-Sequence Learning as Beam-Search Optimizationcited by 126
  8. 2022Contrastive Search Is What You Need For Neural Text Generationcited by 58
  9. 2019Pun Generation with Surprisecited by 37
  10. 2022CoNT: Contrastive Neural Text Generationcited by 21
  11. 2020Decoding Methods for Neural Narrative Generationcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 69; 2024: 164; 2025: 176; 2026: 183.

598 papers.

Sort:

By area:

  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  2. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  3. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  4. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  5. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 367
  6. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 319
  7. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 316
  8. 2024Guiding a Diffusion Model with a Bad Version of Itselfcited by 286
  9. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  10. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 266
  11. 2024Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Textcited by 255
  12. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 233
38 more
  1. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  2. 2024Chain-of-Thought Reasoning Without Promptingcited by 208
  3. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  4. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decodingcited by 198
  5. 2024Multi-Modal Hallucination Control by Visual Information Groundingcited by 196
  6. 2023Does Writing with Language Models Reduce Content Diversity?cited by 190
  7. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  8. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 171
  9. 2023On the Reliability of Watermarks for Large Language Modelscited by 170
  10. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 168
  11. 2024RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectorscited by 157
  12. 2023A Reparameterized Discrete Diffusion Model for Text Generationcited by 152
  13. 2024Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputscited by 148
  14. 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 140
  15. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  16. 2024Rho-1: Not All Tokens Are What You Needcited by 132
  17. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 129
  18. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decodingcited by 118
  19. 2024A Thorough Examination of Decoding Methods in the Era of LLMscited by 112
  20. 2023SCOTT: Self-Consistent Chain-of-Thought Distillationcited by 103
  21. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 100
  22. 2023NEFTune: Noisy Embeddings Improve Instruction Finetuningcited by 99
  23. 2024ARGS: Alignment as Reward-Guided Searchcited by 99
  24. 2023The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Textcited by 97
  25. 2023Amortizing intractable inference in large language modelscited by 94
  26. 2024Tug-of-War Between Knowledge: Exploring and Resolving Knowledge Conflicts in Retrieval-Augmented Language Modelscited by 93
  27. 2023Stay on topic with Classifier-Free Guidancecited by 90
  28. 2024TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Spacecited by 90
  29. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 90
  30. 2024Tuning Language Models by Proxycited by 88
  31. 2024Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Modelscited by 87
  32. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigationcited by 87
  33. 2024Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Differencecited by 85
  34. 2022Training Trajectories of Language Models Across Scalescited by 80
  35. 2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capabilitycited by 80
  36. 2024Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluationcited by 79
  37. 2023Alleviating Hallucinations of Large Language Models through Induced Hallucinationscited by 78
  38. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attentioncited by 74

50 most cited of 598 All 598 (JSON)