mlsanityData to 14 Oct 2026

DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models

arXiv 2309.03883 ↗First posted 7 Sep 2023cs.CL cs.AI cs.LGcited by 376

Not on the map. Why?

Cited by 136 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Despite their impressive capabilities, large language models (LLMs) are prone to hallucinations, i.e., generating content that deviates from facts seen during pretraining. We propose a simple decoding strategy for reducing hallucinations with pretrained LLMs that does not require conditioning on retrieved external knowledge nor additional fine-tuning. Our approach obtains the next-token distribution by contrasting the differences in logits obtained from projecting the later layers versus earlier layers to the vocabulary space, exploiting the fact that factual knowledge in an LLMs has generally been shown to be localized to particular transformer layers. We find that this Decoding by Contrasting Layers (DoLa) approach is able to better surface factual knowledge and reduce the generation of incorrect facts. DoLa consistently improves the truthfulness across multiple choices tasks and open-ended generation tasks, for example improving the performance of LLaMA family models on TruthfulQA by 12-17% absolute points, demonstrating its potential in making LLMs reliably generate truthful facts.

Key facts. Cited by 376 papers in the map (data through 14 Oct 2026). Builds on 34 of its 46 references that are in the map. First posted on arXiv 7 Sep 2023.

Builds on

34 of 46 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  10. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  11. 2022Emergent Abilities of Large Language Modelscited by 2,512
  12. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
22 more
  1. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  2. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  3. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  4. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  5. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  6. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
  7. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  8. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
  9. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 762
  10. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  11. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  12. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  13. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  14. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  15. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 596
  16. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 327
  17. 2022Confident Adaptive Language Modelingcited by 310
  18. 2019Depth-Adaptive Transformercited by 236
  19. 2023Generating Benchmarks for Factuality Evaluation of Language Modelscited by 105
  20. 2022Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generationcited by 94
  21. 2023The Benefits of Bad Advice: Autocontrastive Decoding across Model Layerscited by 29
  22. 2021Not All Models Localize Linguistic Knowledge in the Same Place: A Layer-wise Probing on BERToids' Representationscited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 16; 2024: 93; 2025: 125; 2026: 142.

376 papers.

Sort:

By area:

  1. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 478
  2. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 388
  3. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  4. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 244
  5. 2023Fine-tuning Language Models for Factualitycited by 221
  6. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 206
  7. 2023Can LLM-Generated Misinformation Be Detected?cited by 177
  8. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 174
  9. 2024Large Legal Fictions: Profiling Legal Hallucinations in Large Language Modelscited by 150
  10. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decodingcited by 118
  11. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  12. 2024ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretabilitycited by 101
38 more
  1. 2024HaloScope: Harnessing Unlabeled LLM Generations for Hallucination Detectioncited by 99
  2. 2024Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Eracited by 98
  3. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 94
  4. 2024Tuning Language Models by Proxycited by 89
  5. 2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 89
  6. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  7. 2024Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Differencecited by 88
  8. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigationcited by 88
  9. 2024Unfamiliar Finetuning Examples Control How Language Models Hallucinatecited by 87
  10. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 85
  11. 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Modelscited by 85
  12. 2026Verbalizable Representations Form a Global Workspace in Language Modelscited by 75
  13. 2023Contrastive Decoding Improves Reasoning in Large Language Modelscited by 74
  14. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attentioncited by 74
  15. 2024Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Modelscited by 72
  16. 2024The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalitiescited by 69
  17. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 66
  18. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 65
  19. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  20. 2024Knowledge Circuits in Pretrained Transformerscited by 60
  21. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 60
  22. 2024Spectral Editing of Activations for Large Language Model Alignmentcited by 56
  23. 2024Seeing Clearly by Layer Two: Enhancing Attention Heads to Alleviate Hallucination in LVLMscited by 52
  24. 2024Can AI Assistants Know What They Don't Know?cited by 50
  25. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 50
  26. 2023Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regressioncited by 49
  27. 2025Large Language Models Hallucination: A Comprehensive Surveycited by 49
  28. 2025Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinationscited by 48
  29. 2023Personas as a Way to Model Truthfulness in Language Modelscited by 47
  30. 2024How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Promptscited by 47
  31. 2024FLAME: Factuality-Aware Alignment for Large Language Modelscited by 47
  32. 2024Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projectioncited by 46
  33. 2024In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigationcited by 45
  34. 2025Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Modelscited by 44
  35. 2025Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decodingcited by 43
  36. 2024Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causalitycited by 42
  37. 2023Deficiency of Large Language Models in Finance: An Empirical Examination of Hallucinationcited by 41
  38. 2025Parameters vs. Context: Fine-Grained Control of Knowledge Reliance in Language Modelscited by 39

50 most cited of 376 All 376 (JSON)