Data to 7 Oct 2026

When to Trust LLMs: Aligning Confidence with Response Quality

arXiv 2404.17287 ↗First posted 26 Apr 2024cs.CLcited by 42

Not on the map. Why?

Abstract (arXiv)

Despite the success of large language models (LLMs) in natural language generation, much evidence shows that LLMs may produce incorrect or nonsensical text. This limitation highlights the importance of discerning when to trust LLMs, especially in safety-critical domains. Existing methods often express reliability by confidence level, however, their effectiveness is limited by the lack of objective guidance. To address this, we propose CONfidence-Quality-ORDer-preserving alignment approach (CONQORD), which leverages reinforcement learning guided by a tailored dual-component reward function. This function integrates quality reward and order-preserving alignment reward functions. Specifically, the order-preserving reward incentivizes the model to verbalize greater confidence for responses of higher quality to align the order of confidence and quality. Experiments demonstrate that CONQORD significantly improves the alignment performance between confidence and response accuracy, without causing over-cautious. Furthermore, the aligned confidence provided by CONQORD informs when to trust LLMs, and acts as a determinant for initiating the retrieval process of external knowledge. Aligning confidence with response quality ensures more transparent and reliable responses, providing better trustworthiness.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 23 of its 35 references that are in the map. First posted on arXiv 26 Apr 2024.

Builds on

23 of 35 references.

References · 23

2020Language Models are Few-Shot Learnerscited by 32,199
2017Proximal Policy Optimization Algorithmscited by 14,441
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
2017On Calibration of Modern Neural Networkscited by 4,860
2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
2019Fine-Tuning Language Models from Human Preferencescited by 2,124
2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
2022Teaching Models to Express Their Uncertainty in Wordscited by 731

11 more

2021A Survey of Uncertainty in Deep Neural Networkscited by 675
2023Zephyr: Direct Distillation of LM Alignmentcited by 498
2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
2021Revisiting the Calibration of Modern Neural Networkscited by 387
2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 155
2023Do Language Models Know When They're Hallucinating References?cited by 115
2023Alleviating Hallucinations of Large Language Models through Induced Hallucinationscited by 78
2023MacLaSa: Multi-Aspect Controllable Text Generation via Efficient Sampling from Compact Latent Spacecited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 20; 2026: 16.

42 papers.

Sort:

By area:

2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 105
2025Efficient Test-Time Scaling via Self-Calibrationcited by 74
2025Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Modelscited by 48
2024Knowledge Boundary of Large Language Models: A Surveycited by 36
2025Adaptive Retrieval Without Self-Knowledge? Bringing Uncertainty Back Homecited by 33
2025Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approachcited by 25
2025Extracting Probabilistic Knowledge from Large Language Models for Bayesian Network Parameterizationcited by 15
2025Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missingcited by 14
2024CtrlA: Adaptive Retrieval-Augmented Generation via Inherent Controlcited by 13
2025Enhancing LLM Reliability via Explicit Knowledge Boundary Modelingcited by 13
2025Self-Route: Automatic Mode Switching via Capability Estimation for Efficient Reasoningcited by 8

30 more

2025Modeling Uncertainty Trends for Timely Retrieval in Dynamic RAGcited by 8
2026BaseCal: Unsupervised Confidence Calibration via Base Model Signalscited by 7
2025BrowseConf: Confidence-Guided Test-Time Scaling for Web Agentscited by 5
2025Learning from Reference Answers: Versatile Language Model Alignment without Binary Human Preference Datacited by 4
2025Transparentize the Internal and External Knowledge Utilization in LLMs with Trustworthy Citationcited by 4
2026Calibration-Aware Policy Optimization for Reasoning LLMscited by 4
2026Confidence Calibration for Multimodal LLMs: An Empirical Study through Medical VQAcited by 4
2025Towards Harmonized Uncertainty Estimation for Large Language Modelscited by 3
2025Judging with Confidence: Calibrating Autoraters to Preference Distributionscited by 3
2025Let the Model Distribute Its Doubt: Confidence Estimation through Verbalized Probability Distributioncited by 3
2026ORCE: Order-Aware Alignment of Verbalized Confidence in Large Language Modelscited by 3
2026Closing the Reflection Gap: A Free Calibration Bonus for Agentic RLcited by 3
2024When Not to Answer: Evaluating Prompts on GPT Models for Effective Abstention in Unanswerable Math Word Problemscited by 2
2025REHEARSE: Experiential Rehearsal for Verbal Confidence Calibration in Large Language Modelscited by 2
2026Linking Perception, Confidence and Accuracy in MLLMscited by 2
2026Question Difficulty Estimation for Large Language Models via Answer Plausibility Scoringcited by 2
2026Structuring the Space of Sociotechnical Alignmentcited by 2
2024Evaluating Consistencies in LLM responses through a Semantic Clustering of Question Answeringcited by 1
2025RAGs to Riches: RAG-like Few-shot Learning for Large Language Model Role-playingcited by 1
2025When Distance Distracts: Representation Distance Bias in BT-Loss for Reward Modelscited by 1
2025Direct Confidence Alignment: Aligning Verbalized Confidence with Internal Confidence In Large Language Modelscited by 1
2026Monitoring Web Agents Without Internal Signals: Observable Trajectories and Key-Step Supervisioncited by 1
2025On the Diminishing Returns of Complex Robust RAG Training in the Era of Powerful LLMscited by 0
2026Fact-Checking with Large Language Models via Probabilistic Certainty and Consistencycited by 0
2026From the Inside Out: Progressive Distribution Refinement for Confidence Calibrationcited by 0
2026Answering the Wrong Question: Reasoning Trace Inversion for Abstention in LLMscited by 0
2026Identifying Influential N-grams in Confidence Calibration via Regression Analysiscited by 0
2026From Critic to Confidence: PPO for Language-Based Quantitative Prediction with Confidence Estimationcited by 0
2026Routing Without Training: Controllable-Ratio LLM Offloading via Reliability Gatingcited by 0
2026Detecting Hallucination in LLMs: Tracing the Topological Signatures of Impaired Context Sharingcited by 0

Similar, not cited

Close in wording, no citation either way