mlsanityData to 7 Oct 2026

Distillation Contrastive Decoding: Improving LLMs Reasoning with Contrastive Decoding and Distillation

arXiv 2402.14874 ↗First posted 21 Feb 2024cs.CL cs.AI cs.LGcited by 16

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

We propose a straightforward approach called Distillation Contrastive Decoding (DCD) to enhance the reasoning capabilities of Large Language Models (LLMs) during inference. In contrast to previous approaches that relied on smaller amateur models or analysis of hidden state differences, DCD employs Contrastive Chain-of-thought Prompting and advanced distillation techniques, including Dropout and Quantization. This approach effectively addresses the limitations of Contrastive Decoding (CD), which typically requires both an expert and an amateur model, thus increasing computational resource demands. By integrating contrastive prompts with distillation, DCD obviates the need for an amateur model and reduces memory usage. Our evaluations demonstrate that DCD significantly enhances LLM performance across a range of reasoning benchmarks, surpassing both CD and existing methods in the GSM8K and StrategyQA datasets.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 19 of its 25 references that are in the map. First posted on arXiv 21 Feb 2024.

Builds on

19 of 25 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Contrastive Chain-of-Thought Prompting15 Nov
  2. 2022Large Language Models are Zero-Shot Reasoners24 May
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 16

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2023Qwen Technical Reportcited by 3,273
  7. 2023Mistral 7Bcited by 3,065
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  9. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  10. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  11. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  12. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
4 more
  1. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  2. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 367
  3. 2023Deductive Verification of Chain-of-Thought Reasoningcited by 122
  4. 2023Contrastive Decoding Improves Reasoning in Large Language Modelscited by 71

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 7; 2026: 4.

16 papers.

Sort:

By area:

  1. 2024Programming Refusal with Conditional Activation Steeringcited by 191
  2. 2025Small Models Struggle to Learn from Strong Reasonerscited by 109
  3. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
  4. 2024ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Modelscited by 28
  5. 2025Exploring Hallucination of Large Multimodal Models in Video Understanding: Benchmark, Analysis and Mitigationcited by 20
  6. 2024Unlocking Decoding-time Controllability: Gradient-Free Multi-Objective Alignment with Contrastive Promptscited by 9
  7. 2026Thinking by Subtraction: Confidence-Driven Contrastive Decoding for LLM Reasoningcited by 5
  8. 2025Advancing Decoding Strategies: Enhancements in Locally Typical Sampling for LLMscited by 4
  9. 2026Learning from Emptiness: De-biasing Listwise Rerankers with Content-Agnostic Probability Calibrationcited by 4
  10. 2025LightReasoner: Can Small Language Models Teach Large Language Models Reasoning?cited by 3
  11. 2025Delta Sampling: Data-Free Knowledge Transfer Across Diffusion Modelscited by 2
  12. 2024On the Transformations across Reward Model, Parameter Update, and In-Context Promptcited by 1
4 more
  1. 2025Enhancing Learned Knowledge in LoRA Adapters Through Efficient Contrastive Decoding on Ascend NPUscited by 0
  2. 2025Contrastive Decoding for Synthetic Data Generation in Low-Resource Language Modelingcited by 0
  3. 2026Temporal Guidance for Large Language Modelscited by 0
  4. 2026Beyond Tokens: A Survey on Decoding Methods for Large Language and Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Mixed Distillation Helps Smaller Language Model Better Reasoning
  2. 2026LoopCD: Loop-wise Contrastive Decoding for Improving Reasoning in Looped Language Models
  3. 2024Multilingual Contrastive Decoding via Language-Agnostic Layers Skipping
  4. 2025"Well, Keep Thinking": Enhancing LLM Reasoning with Adaptive Injection Decoding
  5. 2025Towards Understanding Distilled Reasoning Models: A Representational Approach
  6. 2023Effective Distillation of Table-based Reasoning Ability from LLMs
  7. 2025Language Models Can See Better: Visual Contrastive Decoding For LLM Multimodal Reasoning
  8. 2025Deconstructing Long Chain-of-Thought: A Structured Reasoning Optimization Framework for Long CoT Distillation