mlsanityData to 7 Oct 2026

Integrative Decoding: Improve Factuality via Implicit Self-consistency

arXiv 2410.01556 ↗First posted 2 Oct 2024cs.CL cs.AI cs.LGcited by 8

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Self-consistency-based approaches, which involve repeatedly sampling multiple outputs and selecting the most consistent one as the final response, prove to be remarkably effective in improving the factual accuracy of large language models. Nonetheless, existing methods usually have strict constraints on the task format, largely limiting their applicability. In this paper, we present Integrative Decoding (ID), to unlock the potential of self-consistency in open-ended generation tasks. ID operates by constructing a set of inputs, each prepended with a previously sampled response, and then processes them concurrently, with the next token being selected by aggregating of all their corresponding predictions at each decoding step. In essence, this simple approach implicitly incorporates self-consistency in the decoding objective. Extensive evaluation shows that ID consistently enhances factuality over a wide range of language models, with substantial improvements on the TruthfulQA (+11.2%), Biographies (+15.4%) and LongFact (+8.5%) benchmarks. The performance gains amplify progressively as the number of sampled responses increases, indicating the potential of ID to scale up with repeated sampling.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 62 of its 74 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

62 of 74 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters6 Aug
  2. 2024Integrate the Essence and Eliminate the Dross: Fine-Grained Self-Consistency for Free-Form Language Generation2 Jul
  3. 2024Atomic Self-Consistency for Better Long Form Generations21 May
  4. 2024Self-Consistent Decoding for More Factual Open Responses1 Mar
  5. 2023Universal Self-Consistency for Large Language Model Generation29 Nov
  6. 2023Contrastive Decoding Improves Reasoning in Large Language Models17 Sep
  7. 2023Self-Refine: Iterative Refinement with Self-Feedback30 Mar

Other references · 55

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2023Mistral 7Bcited by 3,065
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
38 more
  1. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  2. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  3. 2020Learning to summarize from human feedbackcited by 2,562
  4. 2024Qwen2 Technical Reportcited by 2,076
  5. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  6. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  7. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  8. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  9. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  10. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  11. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  12. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  13. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  14. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  15. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  16. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  17. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  18. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  19. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  20. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  21. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 367
  22. 2020Calibration of Pre-trained Transformerscited by 307
  23. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 246
  24. 2023Do Large Language Models Know What They Don't Know?cited by 238
  25. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  26. 2017Controlling Linguistic Style Aspects in Neural Language Generationcited by 147
  27. 2023Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Modelscited by 147
  28. 2024Long-form factuality in large language modelscited by 146
  29. 2024Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Mapscited by 140
  30. 2023SpecTr: Fast Speculative Decoding via Optimal Transportcited by 129
  31. 2023Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigationcited by 123
  32. 2022Natural Language to Code Translation with Executioncited by 122
  33. 2024TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Spacecited by 90
  34. 2024LUQ: Long-text Uncertainty Quantification for LLMscited by 79
  35. 2023Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistencycited by 46
  36. 2024In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigationcited by 43
  37. 2023Halo: Estimation and Reduction of Hallucinations in Open-Source Weak Large Language Modelscited by 36
  38. 2021Uncertainty-Aware Machine Translation Evaluationcited by 35

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 3.

8 papers.

  1. 2025Attention-guided Self-reflection for Zero-shot Hallucination Detection in Large Language Modelscited by 16
  2. 2025Aligning Large Language Models to Follow Instructions and Hallucinate Less via Effective Data Filteringcited by 12
  3. 2025Active Layer-Contrastive Decoding Reduces Hallucination in Large Language Model Generationcited by 10
  4. 2025Monitoring Decoding: Mitigating Hallucination via Evaluating the Factuality of Partial Response during Generationcited by 4
  5. 2025Place Matters: Comparing LLM Hallucination Rates for Place-Based Legal Queriescited by 2
  6. 2026MACD: Model-Aware Contrastive Decoding via Counterfactual Datacited by 1
  7. 2026From Flat Facts to Sharp Hallucinations: Detecting Stubborn Errors via Gradient Sensitivitycited by 0
  8. 2026Beyond Tokens: A Survey on Decoding Methods for Large Language and Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Mirror-Consistency: Harnessing Inconsistency in Majority Voting
  2. 2024SLED: Self Logits Evolution Decoding for Improving Factuality in Large Language Models
  3. 2026Enhancing reasoning accuracy in large language models during inference time
  4. 2024Entropy Guided Extrapolative Decoding to Improve Factuality in Large Language Models
  5. 2026Listen to the Layers: Mitigating Hallucinations with Inter-Layer Disagreement
  6. 2024Enhancing Language Model Factuality via Activation-Based Confidence Calibration and Guided Decoding
  7. 2024Internal Consistency and Self-Feedback in Large Language Models: A Survey
  8. 2024When is the consistent prediction likely to be a correct prediction?