mlsanityData to 7 Oct 2026

Confidence Improves Self-Consistency in LLMs

arXiv 2502.06233 ↗First posted 10 Feb 2025cs.CL cs.AIcited by 111

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Self-consistency decoding enhances LLMs' performance on reasoning tasks by sampling diverse reasoning paths and selecting the most frequent answer. However, it is computationally expensive, as sampling many of these (lengthy) paths is required to increase the chances that the correct answer emerges as the most frequent one. To address this, we introduce Confidence-Informed Self-Consistency (CISC). CISC performs a weighted majority vote based on confidence scores obtained directly from the model. By prioritizing high-confidence paths, it can identify the correct answer with a significantly smaller sample size. When tested on nine models and four datasets, CISC outperforms self-consistency in nearly all configurations, reducing the required number of reasoning paths by over 40% on average. In addition, we introduce the notion of within-question confidence evaluation, after showing that standard evaluation methods are poor predictors of success in distinguishing correct and incorrect answers to the same question. In fact, the most calibrated confidence method proved to be the least effective for CISC. Lastly, beyond these practical implications, our results and analyses show that LLMs can effectively judge the correctness of their own outputs, contributing to the ongoing debate on this topic.

Key facts. Cited by 111 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 39 of its 50 references that are in the map. First posted on arXiv 10 Feb 2025.

Builds on

39 of 50 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Make Every Penny Count: Difficulty-Adaptive Self-Consistency for Cost-Efficient Reasoning24 Aug
  2. 2024Large Language Models have Intrinsic Self-Correction Ability21 Jun
  3. 2024Small Language Models Need Strong Verifiers to Self-Correct Reasoning26 Apr
  4. 2024Soft Self-Consistency Improves Language Model Agents20 Feb
  5. 2024On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks12 Feb
  6. 2024Escape Sky-high Cost: Early-stopping Self-Consistency for Multi-step Reasoning19 Jan
  7. 2023LLMs cannot find reasoning errors, but can correct them given the error location14 Nov
  8. 2023Large Language Models Cannot Self-Correct Reasoning Yet3 Oct
  9. 2023SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoning1 Aug
  10. 2023Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs19 May
  11. 2023Self-Refine: Iterative Refinement with Self-Feedback30 Mar
  12. 2022Language Models (Mostly) Know What They Know11 Jul

Other references · 27

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2017On Calibration of Modern Neural Networkscited by 4,860
  6. 2023Let's Verify Step by Stepcited by 3,598
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  9. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  10. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  11. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  12. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
15 more
  1. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  2. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  3. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  4. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  5. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 304
  6. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 230
  7. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 227
  8. 2023Fine-tuning Language Models for Factualitycited by 219
  9. 2023Can Large Language Models Really Improve by Self-critiquing Their Own Plans?cited by 107
  10. 2023Answering Questions by Meta-Reasoning over Multiple Chains of Thoughtcited by 87
  11. 2024Calibrating Large Language Models with Sample Consistencycited by 72
  12. 2023Self-Verification Improves Few-Shot Clinical Information Extractioncited by 52
  13. 2023A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoningcited by 41
  14. 2024Direct Evaluation of Chain-of-Thought in Multi-hop Reasoning with Knowledge Graphscited by 32
  15. 2024Finetuning Language Models to Emit Linguistic Expressions of Uncertaintycited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 50; 2026: 61.

111 papers.

Sort:

By area:

  1. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
  2. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  3. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  4. 2025Beyond Binary Rewards: Training LMs to Reason About Their Uncertaintycited by 107
  5. 2025Efficient Reasoning Models: A Surveycited by 93
  6. 2025AgentRxiv: Towards Collaborative Autonomous Researchcited by 64
  7. 2025LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arenacited by 36
  8. 2025Debate Only When Necessary: Adaptive Multiagent Collaboration for Efficient LLM Reasoningcited by 27
  9. 2025Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Rewardcited by 25
  10. 2025From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centeredcited by 21
  11. 2025Trace Length is a Simple Uncertainty Signal in Reasoning Modelscited by 16
  12. 2025Do Repetitions Matter? Strengthening Reliability in LLM Evaluationscited by 14
38 more
  1. 2025Calibrating Verbalized Confidence with Self-Generated Distractorscited by 14
  2. 2025SimuHome: A Temporal- and Environment-Aware Benchmark for Smart Home LLM Agentscited by 13
  3. 2025Mitigating LLM Hallucination via Behaviorally Calibrated Reinforcement Learningcited by 13
  4. 2026Parallel-Probe: Towards Efficient Parallel Thinking via 2D Probingcited by 13
  5. 2025Geometric Uncertainty for Detecting and Correcting Hallucinations in LLMscited by 12
  6. 2025Slim-SC: Thought Pruning for Efficient Scaling with Self-Consistencycited by 12
  7. 2025Exploring the System 1 Thinking Capability of Large Reasoning Modelscited by 10
  8. 2025Harnessing Consistency for Robust Test-Time LLM Ensemblecited by 10
  9. 2025Understanding and Steering the Cognitive Behaviors of Reasoning Models at Test-Timecited by 10
  10. 2026Reliability-Aware Adaptive Self-Consistency for Efficient Sampling in LLM Reasoningcited by 10
  11. 2025PiCSAR: Probabilistic Confidence Selection And Ranking for Reasoning Chainscited by 9
  12. 2025Self-Training Large Language Models with Confident Reasoningcited by 8
  13. 2025Guided by Gut: Efficient Test-Time Scaling with Reinforced Intrinsic Confidencecited by 8
  14. 2025How Far Are We from Optimal Reasoning Efficiency?cited by 8
  15. 2025Optimal Self-Consistency for Efficient Reasoning with Large Language Modelscited by 8
  16. 2026Multi-Agent Debate with Memory Maskingcited by 8
  17. 2026LLMs Improving LLMs: Agentic Discovery for Test-Time Scalingcited by 7
  18. 2025Measuring Reasoning Utility in LLMs via Conditional Entropy Reductioncited by 6
  19. 2025Uncertainty-Aware Answer Selection for Improved Reasoning in Multi-LLM Systemscited by 6
  20. 2025Representation Consistency for Accurate and Coherent LLM Answer Aggregationcited by 5
  21. 2025MedKGent: A Large Language Model Agent Framework for Constructing Temporally Evolving Medical Knowledge Graphcited by 5
  22. 2025AlphaApollo: A System for Deep Agentic Reasoningcited by 5
  23. 2026Reliable Thinking with Imagescited by 5
  24. 2026Thinking by Subtraction: Confidence-Driven Contrastive Decoding for LLM Reasoningcited by 5
  25. 2026BAS: A Decision-Theoretic Approach to Evaluating Large Language Model Confidencecited by 5
  26. 2025Generalization of Fine-Tuned Uncertainty Communication and Metacognition in Large Language Modelscited by 4
  27. 2025Re-FORC: Adaptive Reward Prediction for Efficient Chain-of-Thought Reasoningcited by 4
  28. 2026RACER: Risk-Aware Calibrated Efficient Routing for Large Language Modelscited by 4
  29. 2026GCoT-Decoding: Unlocking Deep Reasoning Paths for Universal Question Answeringcited by 4
  30. 2026Hallucinations Undermine Trust; Metacognition is a Way Forwardcited by 4
  31. 2026Unveiling the Entropy Dynamics of Chain-of-Thought Reasoningcited by 4
  32. 2025GrACE: A Generative Approach to Better Confidence Elicitation and Efficient Test-Time Scaling in Large Language Modelscited by 3
  33. 2025MoEs Are Stronger than You Think: Hyper-Parallel Inference Scaling with RoEcited by 3
  34. 2025Self-Reflective Generation at Test Timecited by 3
  35. 2025Optimal Aggregation of LLM and PRM Signals for Efficient Test-Time Scalingcited by 3
  36. 2025When Bias Pretends to Be Truth: How Spurious Correlations Undermine Hallucination Detection in LLMscited by 3
  37. 2025STaR: Towards Effective and Stable Table Reasoning via Slow-Thinking Large Language Modelscited by 3
  38. 2026CoRefine: Confidence-Guided Self-Refinement for Adaptive Test-Time Computecited by 3

50 most cited of 111 All 111 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Is Discrete Difficulty Sufficient? Leveraging Continuous Difficulty for Efficient Self-Consistency in LLMs
  2. 2026Does Inference Scaling Improve Reasoning Faithfulness? A Multi-Model Analysis of Self-Consistency Tradeoffs
  3. 2024Mirror-Consistency: Harnessing Inconsistency in Majority Voting
  4. 2025Certified Self-Consistency: Statistical Guarantees and Test-Time Training for Reliable Reasoning in LLMs
  5. 2024Reasoning Aware Self-Consistency: Leveraging Reasoning Paths for Efficient LLM Sampling
  6. 2026Unsupervised Confidence Calibration for Reasoning LLMs from a Single Generation
  7. 2024Lachesis: Predicting LLM Inference Accuracy using Structural Properties of Reasoning Paths
  8. 2025Latent Self-Consistency for Reliable Majority-Set Selection in Short- and Long-Answer Reasoning