# Decoding by Contrasting Knowledge: Enhancing LLMs' Confidence on Edited Facts

arXiv: https://arxiv.org/abs/2405.11613 · HTML: https://mlsanity.com/paper/2405.11613/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 34 references that are in the map.
- First posted on arXiv 19 May 2024.

## Builds on (most cited first, 30 of 31)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://mlsanity.com/paper/2309.01219/) (2023)
- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [Contrastive Decoding: Open-ended Text Generation as Optimization](https://mlsanity.com/paper/2210.15097/) (2022)
- [Editing Factual Knowledge in Language Models](https://mlsanity.com/paper/2104.08164/) (2021)
- [Memory-Based Model Editing at Scale](https://mlsanity.com/paper/2206.06520/) (2022)
- [DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models](https://mlsanity.com/paper/2309.03883/) (2023)
- [Editing Large Language Models: Problems, Methods, and Opportunities](https://mlsanity.com/paper/2305.13172/) (2023)
- [MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions](https://mlsanity.com/paper/2305.14795/) (2023)
- [Can We Edit Factual Knowledge by In-Context Learning?](https://mlsanity.com/paper/2305.12740/) (2023)
- [Evaluating the Ripple Effects of Knowledge Editing in Language Models](https://mlsanity.com/paper/2307.12976/) (2023)
- [Modifying Memories in Transformer Models](https://mlsanity.com/paper/2012.00363/) (2020)
- [Editable Neural Networks](https://mlsanity.com/paper/2004.00345/) (2020)
- [EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Models](https://mlsanity.com/paper/2308.07269/) (2023)
- [Combating Misinformation in the Age of LLMs: Opportunities and Challenges](https://mlsanity.com/paper/2311.05656/) (2023)
- [Alleviating Hallucinations of Large Language Models through Induced Hallucinations](https://mlsanity.com/paper/2312.15710/) (2023)
- [A Survey of Knowledge Enhanced Pre-trained Language Models](https://mlsanity.com/paper/2211.05994/) (2022)
- [Understanding and Patching Compositional Reasoning in LLMs](https://mlsanity.com/paper/2402.14328/) (2024)
- [KG-Rank: Enhancing Large Language Models for Medical QA with Knowledge Graphs and Ranking Techniques](https://mlsanity.com/paper/2403.05881/) (2024)
- [DeepEdit: Knowledge Editing as Decoding with Constraints](https://mlsanity.com/paper/2401.10471/) (2024)
- [Retrieval-Generation Synergy Augmented Large Language Models](https://mlsanity.com/paper/2310.05149/) (2023)
- [SLANG: New Concept Comprehension of Large Language Models](https://mlsanity.com/paper/2401.12585/) (2024)
- [Retrieval-enhanced Knowledge Editing in Language Models for Multi-Hop Question Answering](https://mlsanity.com/paper/2403.19631/) (2024)
- [FMint: Bridging Human Designed and Data Pretrained Models for Differential Equation Foundation Model](https://mlsanity.com/paper/2404.14688/) (2024)
- [Comparing Hallucination Detection Metrics for Multilingual Generation](https://mlsanity.com/paper/2402.10496/) (2024)
- [Is Factuality Enhancement a Free Lunch For LLMs? Better Factuality Can Lead to Worse Context-Faithfulness](https://mlsanity.com/paper/2404.00216/) (2024)

## Built on by (most cited first, 30 of 36)

- [From System 1 to System 2: A Survey of Reasoning Large Language Models](https://mlsanity.com/paper/2502.17419/) (2025)
- [Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capability](https://mlsanity.com/paper/2411.19943/) (2024)
- [AnyEdit: Edit Any Knowledge Encoded in Language Models](https://mlsanity.com/paper/2502.05628/) (2025)
- [Context-DPO: Aligning Language Models for Context-Faithfulness](https://mlsanity.com/paper/2412.15280/) (2024)
- [Uncovering Overfitting in Large Language Model Editing](https://mlsanity.com/paper/2410.07819/) (2024)
- [Parameters vs. Context: Fine-Grained Control of Knowledge Reliance in Language Models](https://mlsanity.com/paper/2503.15888/) (2025)
- [Can Knowledge Editing Really Correct Hallucinations?](https://mlsanity.com/paper/2410.16251/) (2024)
- [Can Editing LLMs Inject Harm?](https://mlsanity.com/paper/2407.20224/) (2024)
- [Innate Reasoning is Not Enough: In-Context Learning Enhances Reasoning Large Language Models with Less Overthinking](https://mlsanity.com/paper/2503.19602/) (2025)
- [MSP-MVS: Multi-Granularity Segmentation Prior Guided Multi-View Stereo](https://mlsanity.com/paper/2407.19323/) (2024)
- [HiddenGuard: Fine-Grained Safe Generation with Specialized Representation Router](https://mlsanity.com/paper/2410.02684/) (2024)
- [Adaptive Token Biaser: Knowledge Editing via Biasing Key Entities](https://mlsanity.com/paper/2406.12468/) (2024)
- [a1: Steep Test-time Scaling Law via Environment Augmented Generation](https://mlsanity.com/paper/2504.14597/) (2025)
- [DVP-MVS: Synergize Depth-Edge and Visibility Prior for Multi-View Stereo](https://mlsanity.com/paper/2412.11578/) (2024)
- [ParamMute: Suppressing Knowledge-Critical FFNs for Faithful Retrieval-Augmented Generation](https://mlsanity.com/paper/2502.15543/) (2025)
- [StruEdit: Structured Outputs Enable the Fast and Accurate Knowledge Editing for Large Language Models](https://mlsanity.com/paper/2409.10132/) (2024)
- [Research on the Design of a Short Video Recommendation System Based on Multimodal Information and Differential Privacy](https://mlsanity.com/paper/2504.08751/) (2025)
- [A Survey of Calibration Process for Black-Box LLMs](https://mlsanity.com/paper/2412.12767/) (2024)
- [Bring Your Own Knowledge: A Survey of Methods for LLM Knowledge Expansion](https://mlsanity.com/paper/2502.12598/) (2025)
- [What if Agents Could Imagine? Reinforcing Open-Vocabulary HOI Comprehension through Generation](https://mlsanity.com/paper/2602.11499/) (2026)
- [GeoEdit: Geometric Knowledge Editing for Large Language Models](https://mlsanity.com/paper/2502.19953/) (2025)
- [Large Language Models as Computable Approximations to Solomonoff Induction](https://mlsanity.com/paper/2505.15784/) (2025)
- [Rethinking All Evidence: Enhancing Trustworthy Retrieval-Augmented Generation via Conflict-Driven Summarization](https://mlsanity.com/paper/2507.01281/) (2025)
- [A Survey on Proactive Defense Strategies Against Misinformation in Large Language Models](https://mlsanity.com/paper/2507.05288/) (2025)
- [Credit Risk Identification in Supply Chains Using Generative Adversarial Networks](https://mlsanity.com/paper/2501.10348/) (2025)
- [LM-PUB-QUIZ: A Comprehensive Framework for Zero-Shot Evaluation of Relational Knowledge in Language Models](https://mlsanity.com/paper/2408.15729/) (2024)
- [Tutorial on Using Machine Learning and Deep Learning Models for Mental Illness Detection](https://mlsanity.com/paper/2502.04342/) (2025)
- [Fact or Guesswork? Evaluating Large Language Models' Medical Knowledge with Structured One-Hop Judgments](https://mlsanity.com/paper/2502.14275/) (2025)
- [IndusAgent: Reinforcing Open-Vocabulary Industrial Anomaly Detection with Agentic Tools](https://mlsanity.com/paper/2605.20682/) (2026)
- [PFME: A Modular Approach for Fine-grained Hallucination Detection and Editing of Large Language Models](https://mlsanity.com/paper/2407.00488/) (2024)

## Abstract (arXiv)

The knowledge within large language models (LLMs) may become outdated quickly. While in-context editing (ICE) is currently the most effective method for knowledge editing (KE), it is constrained by the black-box modeling of LLMs and thus lacks interpretability. Our work aims to elucidate the superior performance of ICE on the KE by analyzing the impacts of in-context new knowledge on token-wise distributions. We observe that despite a significant boost in logits of the new knowledge, the performance of is still hindered by stubborn knowledge. Stubborn knowledge refers to as facts that have gained excessive confidence during pretraining, making it hard to edit effectively. To address this issue and further enhance the performance of ICE, we propose a novel approach termed Decoding by Contrasting Knowledge (DeCK). DeCK derives the distribution of the next token by contrasting the logits obtained from the newly edited knowledge guided by ICE with those from the unedited parametric knowledge. Our experiments consistently demonstrate that DeCK enhances the confidence of LLMs in edited facts. For instance, it improves the performance of LLaMA3-8B-instruct on MQuAKE by up to 219%, demonstrating its capability to strengthen ICE in the editing of stubborn knowledge. Our work paves the way to develop the both effective and accountable KE methods for LLMs. (The source code is available at: https://deck-llm.meirtz.com)