mlsanityData to 7 Oct 2026

Too Consistent to Detect: A Study of Self-Consistent Errors in LLMs

arXiv 2505.17656 ↗First posted 23 May 2025cs.CLcited by 18

Not on the map. Why?

Abstract (arXiv)

As large language models (LLMs) often generate plausible but incorrect content, error detection has become increasingly critical to ensure truthfulness. However, existing detection methods often overlook a critical problem we term as self-consistent error, where LLMs repeatedly generate the same incorrect response across multiple stochastic samples. This work formally defines self-consistent errors and evaluates mainstream detection methods on them. Our investigation reveals two key findings: (1) Unlike inconsistent errors, whose frequency diminishes significantly as the LLM scale increases, the frequency of self-consistent errors remains stable or even increases. (2) All four types of detection methods significantly struggle to detect self-consistent errors. These findings reveal critical limitations in current detection methods and underscore the need for improvement. Motivated by the observation that self-consistent errors often differ across LLMs, we propose a simple but effective cross-model probe method that fuses hidden state evidence from an external verifier LLM. Our method significantly enhances performance on self-consistent errors across three LLM families.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 24 of its 35 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

24 of 35 references.

References · 24

  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  2. 2024Qwen2.5 Technical Reportcited by 4,713
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  4. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  5. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  6. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  7. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  8. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  9. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  10. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  11. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  12. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
12 more
  1. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 525
  2. 2024Measuring short-form factuality in large language modelscited by 345
  3. 2024INSIDE: LLMs' Internal States Retain the Power of Hallucination Detectioncited by 317
  4. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 230
  5. 2020On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translationcited by 127
  6. 2024HaloScope: Harnessing Unlabeled LLM Generations for Hallucination Detectioncited by 96
  7. 2023SAC3: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistencycited by 89
  8. 2024A Survey on Large Language Models for Critical Societal Domains: Finance, Healthcare, and Lawcited by 72
  9. 2024Factual Confidence of LLMs: on Reliability and Robustness of Current Estimatorscited by 59
  10. 2024InternalInspector I2: Robust Confidence Estimation in LLMs through Internal Statescited by 29
  11. 2025Verify when Uncertain: Beyond Self-Consistency in Black Box Hallucination Detectioncited by 18
  12. 2024PoLLMgraph: Unraveling Hallucinations in Large Language Models via State Transition Dynamicscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 17.

18 papers.

Sort:

By area:

  1. 2026CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolutioncited by 9
  2. 2026BaseCal: Unsupervised Confidence Calibration via Base Model Signalscited by 7
  3. 2026Guideline-Grounded Evidence Accumulation for High-Stakes Agent Verificationcited by 4
  4. 2025GrACE: A Generative Approach to Better Confidence Elicitation and Efficient Test-Time Scaling in Large Language Modelscited by 3
  5. 2026CoAct: Co-Active LLM Preference Learning with Human-AI Synergycited by 3
  6. 2026When Self-Consistency Backfires: Majority Vote Hurts the Majority of Hard Science Problems for Small LLMscited by 3
  7. 2026Illusions of Confidence? Diagnosing LLM Truthfulness via Neighborhood Consistencycited by 2
  8. 2026From Signals to Transfer: A Factorised Study of Probe-Based Uncertainty Estimation in Large Language Modelscited by 2
  9. 2026Learning Uncertainty from Sequential Internal Dispersion in Large Language Modelscited by 1
  10. 2026More Correct Mass, Worse Answers: Why Power Sampling Can Fail and How to Fix Itcited by 1
  11. 2026Towards Reliable Truth-Aligned Uncertainty Estimation in Large Language Modelscited by 0
  12. 2026Why Don't You Know? Evaluating the Impact of Uncertainty Sources on Uncertainty Quantification in LLMscited by 0
6 more
  1. 2026Logical Consistency as a Bridge: Improving LLM Hallucination Detection via Label Constraint Modeling between Responses and Self-Judgmentscited by 0
  2. 2026Max-pooling Network Revisited: Analyzing the Role of Semantic Probability in Multiple Instance Learning for Hallucination Detectioncited by 0
  3. 2026Resample or Reroute? Recoverable Stopping Debt Without Identified Action Selectioncited by 0
  4. 2026Decomposing Wrong-Consensus Agreement in LLM Self-Consistencycited by 0
  5. 2026Can VLMs Reliably Assess Sidewalk Accessibility Attributes from Pedestrian-Level Imagery?cited by 0
  6. 2026Reasoning Concentrates Errors, and Self-Consistency Never Noticescited by 0