mlsanityData to 7 Oct 2026

SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationales

arXiv 2405.20974 ↗First posted 31 May 2024cs.CL cs.AI cs.LGcited by 102

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) often generate inaccurate or fabricated information and generally fail to indicate their confidence, which limits their broader applications. Previous work elicits confidence from LLMs by direct or self-consistency prompting, or constructing specific datasets for supervised finetuning. The prompting-based approaches have inferior performance, and the training-based approaches are limited to binary or inaccurate group-level confidence estimates. In this work, we present the advanced SaySelf, a training framework that teaches LLMs to express more accurate fine-grained confidence estimates. In addition, beyond the confidence scores, SaySelf initiates the process of directing LLMs to produce self-reflective rationales that clearly identify gaps in their parametric knowledge and explain their uncertainty. This is achieved by using an LLM to automatically summarize the uncertainties in specific knowledge via natural language. The summarization is based on the analysis of the inconsistency in multiple sampled reasoning chains, and the resulting data is utilized for supervised fine-tuning. Moreover, we utilize reinforcement learning with a meticulously crafted reward function to calibrate the confidence estimates, motivating LLMs to deliver accurate, high-confidence predictions and to penalize overconfidence in erroneous outputs. Experimental results in both in-distribution and out-of-distribution datasets demonstrate the effectiveness of SaySelf in reducing the confidence calibration error and maintaining the task performance. We show that the generated self-reflective rationales are reasonable and can further contribute to the calibration. The code is made public at https://github.com/xu1868/SaySelf.

Key facts. Cited by 102 papers in the map (data through 7 Oct 2026). Builds on 49 of its 75 references that are in the map. First posted on arXiv 31 May 2024.

Builds on

49 of 75 references.

References · 49

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2017On Calibration of Modern Neural Networkscited by 4,860
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  6. 2023Mistral 7Bcited by 3,065
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  10. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  11. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  12. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
37 more
  1. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  2. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  3. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  4. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  5. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  6. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  7. 2023Instruction Tuning with GPT-4cited by 676
  8. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  9. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  10. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  11. 2023A Survey of Hallucination in Large Foundation Modelscited by 365
  12. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  13. 2023Explainability for Large Language Models: A Surveycited by 331
  14. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  15. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 329
  16. 2023Faithful Chain-of-Thought Reasoningcited by 285
  17. 2023AI Alignment: A Comprehensive Surveycited by 256
  18. 2023A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generationcited by 202
  19. 2023Sources of Hallucination by Large Language Models on Inference Taskscited by 168
  20. 2023A Categorical Archive of ChatGPT Failurescited by 167
  21. 2022On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?cited by 137
  22. 2023Do Language Models Know When They're Hallucinating References?cited by 115
  23. 2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
  24. 2023Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Modelscited by 91
  25. 2023Cognitive Mirage: A Review of Hallucinations in Large Language Modelscited by 83
  26. 2024Calibrating Large Language Models with Sample Consistencycited by 72
  27. 2023Knowledge of Knowledge: Exploring Known-Unknowns Uncertainty with Large Language Modelscited by 60
  28. 2022A Close Look into the Calibration of Pre-trained Language Modelscited by 57
  29. 2023Won't Get Fooled Again: Answering Questions with False Premisescited by 47
  30. 2023Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Modelscited by 45
  31. 2024Learning to Trust Your Feelings: Leveraging Self-awareness in LLMs for Hallucination Mitigationcited by 43
  32. 2021Local Interpretations for Explainable Natural Language Processing: A Surveycited by 31
  33. 2024Chain-of-Thought Unfaithfulness as Disguised Accuracycited by 27
  34. 2017Automatic Generation of Natural Language Explanationscited by 22
  35. 2021Few-Shot Out-of-Domain Transfer Learning of Natural Language Explanations in a Label-Abundant Setupcited by 16
  36. 2023Augmenting LLMs with Knowledge: A survey on hallucination preventioncited by 16
  37. 2024Chain of Thought Explanation for Dialogue State Trackingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 51; 2026: 35.

102 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  2. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
  3. 2025Beyond Binary Rewards: Training LMs to Reason About Their Uncertaintycited by 107
  4. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 105
  5. 2025Maximizing Confidence Alone Improves Reasoningcited by 97
  6. 2025Efficient Test-Time Scaling via Self-Calibrationcited by 74
  7. 2025Reasoning Models Better Express Their Confidencecited by 63
  8. 2024Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Modelscited by 55
  9. 2024Scaling Laws for Predicting Downstream Performance in LLMscited by 48
  10. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 45
  11. 2025Large Language Models Hallucination: A Comprehensive Surveycited by 45
  12. 2025ConfTuner: Training Large Language Models to Express Their Confidence Verballycited by 37
38 more
  1. 2025Position: Uncertainty Quantification Needs Reassessment for Large-language Model Agentscited by 27
  2. 2025Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Predictioncited by 26
  3. 2025When One LLM Drools, Multi-LLM Collaboration Rulescited by 25
  4. 2025Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approachcited by 25
  5. 2025Primus: A Pioneering Collection of Open-Source Datasets for Cybersecurity LLM Trainingcited by 23
  6. 2025Position: Agent Should Invoke External Tools ONLY When Epistemically Necessarycited by 22
  7. 2024A Survey on the Honesty of Large Language Modelscited by 21
  8. 2025TruthRL: Incentivizing Truthful LLMs via Reinforcement Learningcited by 20
  9. 2024LoGU: Long-form Generation with Uncertainty Expressionscited by 19
  10. 2024AnalogXpert: Automating Analog Topology Synthesis by Incorporating Circuit Design Expertise into Large Language Modelscited by 17
  11. 2025LoVeC: Reinforcement Learning for Better Verbalized Confidence in Long-Form Generationscited by 17
  12. 2025Seeing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Modelscited by 16
  13. 2024Towards Rationality in Language and Multimodal Agents: A Surveycited by 15
  14. 2025Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missingcited by 14
  15. 2024DebUnc: Improving Large Language Model Agent Communication With Uncertainty Metricscited by 13
  16. 2026Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theorycited by 13
  17. 2026Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewardscited by 13
  18. 2025Object-Level Verbalized Confidence Calibration in Vision-Language Models via Semantic Perturbationcited by 11
  19. 2025UNCLE: Benchmarking Uncertainty Expressions in Long-Form Generationcited by 11
  20. 2025MMBoundary: Advancing MLLM Knowledge Boundary Awareness through Reasoning Step Confidence Calibrationcited by 11
  21. 2024MACAROON: Training Vision-Language Models To Be Your Engaged Partnerscited by 10
  22. 2025Antidote: A Unified Framework for Mitigating LVLM Hallucinations in Counterfactual Presupposition and Object Perceptioncited by 10
  23. 2025EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solvingcited by 10
  24. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  25. 2024Cross-Refine: Improving Natural Language Explanation Generation by Learning in Tandemcited by 8
  26. 2025BARREL: Boundary-Aware Reasoning for Factual and Reliable LRMscited by 8
  27. 2025SPARTA ALIGNMENT: Collectively Aligning Multiple Language Models through Combatcited by 8
  28. 2025C2GSPG: Confidence-calibrated Group Sequence Policy Gradient towards Self-aware Reasoningcited by 8
  29. 2025MedReflect: Teaching Medical LLMs to Self-Improve via Reflective Correctioncited by 8
  30. 2025Beyond Accuracy: The Role of Calibration in Self-Improving Large Language Modelscited by 7
  31. 2025Solar Photovoltaic Assessment with Large Language Modelcited by 7
  32. 2025Teaching Language Models to Faithfully Express their Uncertaintycited by 7
  33. 2024UBench: Benchmarking Uncertainty in Large Language Models with Multiple Choice Questionscited by 6
  34. 2024FactTest: Factuality Testing in Large Language Models with Finite-Sample and Distribution-Free Guaranteescited by 6
  35. 2025Evaluating Evaluation Metrics -- The Mirage of Hallucination Detectioncited by 6
  36. 2025SelfReflect: Can LLMs Communicate Their Internal Answer Distribution?cited by 6
  37. 2025Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Modelscited by 6
  38. 2026Reaching Beyond the Mode: RL for Distributional Reasoning in Language Modelscited by 6

50 most cited of 102 All 102 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing Uncertainty Estimation in LLMs with Expectation of Aggregated Internal Belief
  2. 2026Understanding and Mitigating Premature Confidence for Better LLM Reasoning
  3. 2026When Words Fall Short: Iterative Synergy Between Verbalized Reasoning and Hidden Features for LLM Confidence Estimation
  4. 2026Know When You're Wrong: Aligning Confidence with Correctness for LLM Error Detection
  5. 2025SteerConf: Steering LLMs for Confidence Elicitation
  6. 2026PRISM: A Unified Framework for Post-Training LLMs Without Verifiable Rewards
  7. 2025Lexical Hints of Accuracy in LLM Reasoning Chains
  8. 2025Self-Training Large Language Models with Confident Reasoning