mlsanityData to 7 Oct 2026

Graph-based Confidence Calibration for Large Language Models

arXiv 2411.02454 ↗First posted 3 Nov 2024cs.CL cs.AI cs.IRcited by 10

Not on the map. Why?

Abstract (arXiv)

Reliable confidence estimation is essential for enhancing the trustworthiness of large language models (LLMs), especially in high-stakes scenarios. Despite its importance, accurately estimating confidence in LLM responses remains a significant challenge. In this work, we propose using an auxiliary learning model to assess response correctness based on the self-consistency of multiple outputs generated by the LLM. Our method builds a consistency graph to represent the agreement among multiple responses and uses a graph neural network (GNN) to estimate the likelihood that each response is correct. Experiments demonstrate that this method has strong calibration performance on various benchmark datasets and generalizes well to out-of-domain cases.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 37 of its 52 references that are in the map. First posted on arXiv 3 Nov 2024.

Builds on

37 of 52 references.

References · 37

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  7. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  8. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  9. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  10. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  11. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  12. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
25 more
  1. 2018CoQA: A Conversational Question Answering Challengecited by 674
  2. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  3. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 286
  4. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 209
  5. 2022Out-of-Distribution Detection and Selective Generation for Conditional Language Modelscited by 193
  6. 2023LM-Polygraph: Uncertainty Estimation for Language Modelscited by 130
  7. 2024Benchmarking LLMs via Uncertainty Quantificationcited by 124
  8. 2023Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthinesscited by 121
  9. 2023A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writingcited by 98
  10. 2023WebCPM: Interactive Web Search for Chinese Long-form Question Answeringcited by 93
  11. 2022Evaluating the Factual Consistency of Large Language Models Through News Summarizationcited by 90
  12. 2024LUQ: Long-text Uncertainty Quantification for LLMscited by 79
  13. 2023A Survey on Uncertainty Quantification Methods for Deep Learningcited by 78
  14. 2024Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approachcited by 73
  15. 2023Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedbackcited by 71
  16. 2024Calibrating Large Language Models Using Their Generations Onlycited by 63
  17. 2024MARS: Meaning-Aware Response Scoring for Uncertainty Estimation in Generative LLMscited by 45
  18. 2023How Ready are Pre-trained Abstractive Models and LLMs for Legal Case Judgement Summarization?cited by 43
  19. 2024Uncertainty in Language Models: Assessment through Rank-Calibrationcited by 40
  20. 2022Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcitycited by 24
  21. 2019AlphaClean: Automatic Generation of Data Cleaning Pipelinescited by 23
  22. 2024LLMClean: Context-Aware Tabular Data Cleaning via LLM-Generated OFDscited by 19
  23. 2024Reconfidencing LLMs from the Grouping Loss Perspectivecited by 11
  24. 2024Can AI Write Classical Chinese Poetry like Humans? An Empirical Study Inspired by Turing Testcited by 7
  25. 2023Methods to Estimate Large Language Model Confidencecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 4.

10 papers.

  1. 2025Two Heads are Better Than One: Test-time Scaling of Multi-agent Collaborative Reasoningcited by 29
  2. 2025From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centeredcited by 21
  3. 2025Calibrating Uncertainty Quantification of Multi-Modal LLMs using Groundingcited by 8
  4. 2025Self-Evaluating LLMs for Multi-Step Tasks: Stepwise Confidence Estimation for Failure Detectioncited by 6
  5. 2025BrowseConf: Confidence-Guided Test-Time Scaling for Web Agentscited by 5
  6. 2025All Roads Lead to Rome: Graph-Based Confidence Estimation for Large Language Model Reasoningcited by 3
  7. 2026LLM Router: Rethinking Routing with Prefill Activationscited by 2
  8. 2026A Systematic Evaluation of Black-Box Uncertainty Estimation Methods for Large Language Modelscited by 2
  9. 2026Completing Missing Annotation: Multi-Agent Debate for Accurate and Scalable Relevant Assessment for IR Benchmarkscited by 1
  10. 2026When Does Delegation Beat Majority? A Delegation-Based Aggregator for Multi-Sample LLM Inferencecited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Calibrating the Confidence of Large Language Models by Eliciting Fidelity
  2. 2026ConfidenceBench: Evaluating Confidence Calibration in Large Language Models
  3. 2025Direct Confidence Alignment: Aligning Verbalized Confidence with Internal Confidence In Large Language Models
  4. 2024Calibrating Long-form Generations from Large Language Models
  5. 2024Confidence Under the Hood: An Investigation into the Confidence-Probability Alignment in Large Language Models
  6. 2026Know When You're Wrong: Aligning Confidence with Correctness for LLM Error Detection
  7. 2026Instinct vs. Reflection: Unifying Token and Verbalized Confidence in Multimodal Large Models
  8. 2025SteerConf: Steering LLMs for Confidence Elicitation