DECK: A Consistency x Confidence Taxonomy of LLM Hallucinations
Not on the map. Why?
Abstract (arXiv)
Existing hallucination taxonomies classify LLM errors by what is wrong with the output -- memorised misconceptions, reasoning failures, fluent fabrications -- but cannot answer a different question: which uncertainty scorer would have caught this error? We propose a complementary taxonomy that classifies errors by their detectability signature, the signal a scorer family would read. The DECK taxonomy is a 2x2 partition along inter-sample consistency and token-level confidence into four regimes (Drift, Entrenched, Confabulation, Knotted) that yields a falsifiable blind-spot map: black-box consistency scorers have signal in D and C, white-box token-probability scorers in K and C, and only an LLM-as-a-Judge with independent pretraining can detect E. Across three models and four short-form QA datasets we test this map two ways: judge-involving scorer disagreements concentrate in each family's predicted blind-spot cells, and external labels (SelfAware unanswerable, HaluEval adversarial, PopQA entity popularity) land in the predicted cells, robustly to cross-fitted thresholds. We further identify a universal blind spot of output-level UQ: on knowledge-gap inputs where the generator emits confident, repeatable fabrications, every output-level family collapses by construction. A linear probe on Llama-3-8B's final-layer hidden states also falls to chance, with or without quantisation, though an intermediate layer retains weak signal.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 17 of its 32 references that are in the map. First posted on arXiv 1 Jun 2026.
Builds on
17 of 32 references.
References · 17
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
- 2022Language Models (Mostly) Know What They Knowcited by 1,856
- 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
- 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
- 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
- 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 913
- 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
- 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
- 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
- 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 180
5 more
- 2024To Believe or Not to Believe Your LLMcited by 120
- 2021Truthful AI: Developing and governing AI that does not liecited by 113
- 2024LUQ: Long-text Uncertainty Quantification for LLMscited by 81
- 2025A Head to Predict and a Head to Question: Pre-trained Uncertainty Quantification Heads for Hallucination Detection in LLM Outputscited by 21
- 2025Uncertainty Quantification for Language Models: A Suite of Black-Box, White-Box, LLM Judge, and Ensemble Scorerscited by 18
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations
- 2025Trust Me, I'm Wrong: LLMs Hallucinate with Certainty Despite Knowing the Answer
- 2026Anchored Confabulation: Partial Evidence Non-Monotonically Amplifies Confident Hallucination in LLMs
- 2026Rethinking Hallucinations: Correctness, Consistency, and Prompt Multiplicity
- 2026HalluGuard: Demystifying Data-Driven and Reasoning-Driven Hallucinations in LLMs
- 2026Logical Consistency as a Bridge: Improving LLM Hallucination Detection via Label Constraint Modeling between Responses and Self-Judgments
- 2026PRISM: Probing Reasoning, Instruction, and Source Memory in LLM Hallucinations
- 2026Hallucinations Undermine Trust; Metacognition is a Way Forward