mlsanityData to 14 Oct 2026

UAlign: Leveraging Uncertainty Estimations for Factuality Alignment on Large Language Models

arXiv 2412.11803 ↗First posted 16 Dec 2024cs.CLcited by 13

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Despite demonstrating impressive capabilities, Large Language Models (LLMs) still often struggle to accurately express the factual knowledge they possess, especially in cases where the LLMs' knowledge boundaries are ambiguous. To improve LLMs' factual expressions, we propose the UAlign framework, which leverages Uncertainty estimations to represent knowledge boundaries, and then explicitly incorporates these representations as input features into prompts for LLMs to Align with factual knowledge. First, we prepare the dataset on knowledge question-answering (QA) samples by calculating two uncertainty estimations, including confidence score and semantic entropy, to represent the knowledge boundaries for LLMs. Subsequently, using the prepared dataset, we train a reward model that incorporates uncertainty estimations and then employ the Proximal Policy Optimization (PPO) algorithm for factuality alignment on LLMs. Experimental results indicate that, by integrating uncertainty representations in LLM alignment, the proposed UAlign can significantly enhance the LLMs' capacities to confidently answer known questions and refuse unknown questions on both in-domain and out-of-domain tasks, showing reliability improvements and good generalizability over various prompt- and training-based baselines.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 50 of its 70 references that are in the map. First posted on arXiv 16 Dec 2024.

Builds on

50 of 70 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluation14 Feb
  2. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback24 May
  3. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 47

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2017On Calibration of Modern Neural Networkscited by 4,960
  6. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  7. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
35 more
  1. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  2. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  3. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,485
  4. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  5. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  6. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 819
  7. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  8. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  9. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  10. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
  11. 2024Knowledge Conflicts for LLMs: A Surveycited by 283
  12. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 232
  13. 2023Fine-tuning Language Models for Factualitycited by 221
  14. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 210
  15. 2023A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generationcited by 206
  16. 2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 159
  17. 2023Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentationcited by 147
  18. 2018Correcting Length Bias in Neural Machine Translationcited by 120
  19. 2022Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysiscited by 104
  20. 2023Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Modelscited by 91
  21. 2024LLM Internal States Reveal Hallucination Risk Faced With a Querycited by 82
  22. 2024Calibrating Large Language Models with Sample Consistencycited by 73
  23. 2024Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedbackcited by 66
  24. 2024CoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreferencecited by 51
  25. 2024Learning to Trust Your Feelings: Leveraging Self-awareness in LLMs for Hallucination Mitigationcited by 44
  26. 2024Benchmarking Knowledge Boundary for Large Language Models: A Different Perspective on Model Evaluationcited by 38
  27. 2023Examining LLMs' Uncertainty Expression Towards Questions Outside Parametric Knowledgecited by 32
  28. 2024Self-DC: When to Reason and When to Act? Self Divide-and-Conquer for Compositional Unknown Questionscited by 30
  29. 2023Improving the Reliability of Large Language Models by Leveraging Uncertainty-Aware In-Context Learningcited by 29
  30. 2023Enhancing Large Language Models Against Inductive Instructions with Dual-critique Promptingcited by 21
  31. 2024Enhancing Confidence Expression in Large Language Models Through Learning from Past Experiencecited by 21
  32. 2023Improving Factual Consistency for Knowledge-Grounded Dialogue Systems via Knowledge Enhancement and Alignmentcited by 19
  33. 2024PopALM: Popularity-Aligned Language Models for Social Media Trendy Response Predictioncited by 8
  34. 2022CoP: Factual Inconsistency Detection by Controlling the Preferencecited by 4
  35. 2022Bayesian Neural Network Language Modeling for Speech Recognitioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 5.

13 papers.

Sort:

By area:

  1. 2025Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Modelscited by 36
  2. 2025TruthRL: Incentivizing Truthful LLMs via Reinforcement Learningcited by 27
  3. 2025Enhancing LLM Reliability via Explicit Knowledge Boundary Modelingcited by 13
  4. 2025BARREL: Boundary-Aware Reasoning for Factual and Reliable LRMscited by 8
  5. 2025ReliableMath: Benchmark of Reliable Mathematical Reasoning on Large Language Modelscited by 8
  6. 2025MR-Align: Meta-Reasoning Informed Factuality Alignment for Large Reasoning Modelscited by 4
  7. 2025Self-Error-Instruct: Generalizing from Errors for LLMs Mathematical Reasoningcited by 2
  8. 2026From Passive Metric to Active Signal: The Evolving Role of Uncertainty Quantification in Large Language Modelscited by 2
  9. 2025Uncertainty-Aware Collaborative System of Large and Small Models for Multimodal Sentiment Analysiscited by 1
  10. 2026GRADE: Probing Knowledge Gaps in LLMs through Gradient Subspace Dynamicscited by 1
  11. 2026LANCET: Neural Intervention via Structural Entropy for Mitigating Faithfulness Hallucinations in LLMscited by 0
  12. 2026PRISM: Probability Reallocation with In-Span Masking for Knowledge-Sensitive Alignmentcited by 0
1 more
  1. 2026Only Say What You Know: Calibration-Aware Generation for Long-Form Factualitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024FactAlign: Long-form Factuality Alignment of Large Language Models
  2. 2026Uncertainty-Aware Abstention in Large Language Models with Provable Alignment Guarantees
  3. 2026FAITH: Factuality Alignment through Integrating Trustworthiness and Honestness
  4. 2026ORCE: Order-Aware Alignment of Verbalized Confidence in Large Language Models
  5. 2026SAGE: Answer-Conditioned Uncertainty Targets for Verbal Uncertainty Alignment
  6. 2024Uncertainty Aware Learning for Language Model Alignment
  7. 2025Revisiting Uncertainty Estimation and Calibration of Large Language Models
  8. 2025An Uncertainty-Driven Adaptive Self-Alignment Framework for Large Language Models