mlsanityData to 14 Oct 2026

Uncertainty Aware Learning for Language Model Alignment

arXiv 2406.04854 ↗First posted 7 Jun 2024cs.CLcited by 13

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

As instruction-tuned large language models (LLMs) evolve, aligning pretrained foundation models presents increasing challenges. Existing alignment strategies, which typically leverage diverse and high-quality data sources, often overlook the intrinsic uncertainty of tasks, learning all data samples equally. This may lead to suboptimal data efficiency and model performance. In response, we propose uncertainty-aware learning (UAL) to improve the model alignment of different task scenarios, by introducing the sample uncertainty (elicited from more capable LLMs). We implement UAL in a simple fashion -- adaptively setting the label smoothing value of training according to the uncertainty of individual samples. Analysis shows that our UAL indeed facilitates better token clustering in the feature space, validating our hypothesis. Extensive experiments on widely used benchmarks demonstrate that our UAL significantly and consistently outperforms standard supervised fine-tuning. Notably, LLMs aligned in a mixed scenario have achieved an average improvement of 10.62% on high-entropy tasks (i.e., AlpacaEval leaderboard), and 1.81% on complex low-entropy tasks (i.e., MetaMath and GSM8K).

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 36 references that are in the map. First posted on arXiv 7 Jun 2024.

Builds on

24 of 36 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning25 Dec
  2. 2023LIMA: Less Is More for Alignment18 May

Other references · 22

  1. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  4. 2019When Does Label Smoothing Help?cited by 988
  5. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  6. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  7. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  8. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 560
  9. 2018Predictive Uncertainty Estimation via Prior Networkscited by 557
  10. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  11. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 338
  12. 2021In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learningcited by 301
10 more
  1. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 229
  2. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
  3. 2014A Tutorial on Principal Component Analysiscited by 141
  4. 2023Towards Making the Most of ChatGPT for Machine Translationcited by 138
  5. 2023Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Modelscited by 87
  6. 2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimismcited by 69
  7. 2023Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Taskscited by 53
  8. 2022Fast Multi-view Clustering via Ensembles: Towards Scalability, Superiority, and Simplicitycited by 28
  9. 2024Take Care of Your Prompt Bias! Investigating and Mitigating Prompt Bias in Factual Knowledge Extractioncited by 13
  10. 2023Self-Evolution Learning for Discriminative Language Model Pretrainingcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 7; 2026: 4.

13 papers.

Sort:

By area:

  1. 2025TruthRL: Incentivizing Truthful LLMs via Reinforcement Learningcited by 27
  2. 2024CEB: Compositional Evaluation Benchmark for Fairness in Large Language Modelscited by 18
  3. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 15
  4. 2025GRAIT: Gradient-Driven Refusal-Aware Instruction Tuning for Effective Hallucination Mitigationcited by 11
  5. 2025Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Modelscited by 7
  6. 2024mathbbUSCD: Improving Code Generation of LLMs by Uncertainty-Aware Selective Contrastive Decodingcited by 6
  7. 2025Uncertainty-o: One Model-agnostic Framework for Unveiling Uncertainty in Large Multimodal Modelscited by 5
  8. 2025Self-Filtered Distillation with LLMs-generated Trust Indicators for Reliable Patent Classificationcited by 4
  9. 2026Conformal Feedback Alignment: Quantifying Answer-Level Reliability for Robust LLM Alignmentcited by 4
  10. 2026UAT-LITE: Inference-Time Uncertainty-Aware Attention for Pretrained Transformerscited by 4
  11. 2025Robust and Generalizable GNN Fine-Tuning via Uncertainty-aware Adapter Learningcited by 0
  12. 2026Reasoning Portability: Guiding Continual Learning for MLLMs in the RLVR Eracited by 0
1 more
  1. 2026Code-MUE: Measuring Code LLMs' Uncertainty through Execution-based Semantic Interaction Graphscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025An Uncertainty-Driven Adaptive Self-Alignment Framework for Large Language Models
  2. 2024UAlign: Leveraging Uncertainty Estimations for Factuality Alignment on Large Language Models
  3. 2026Trust Your Guide Only When Certain: Uncertainty-Aware Sparse Alignment at Inference Time
  4. 2025Revisiting Uncertainty Estimation and Calibration of Large Language Models
  5. 2026Sensitivity Uncertainty Alignment in Large Language Models ↗
  6. 2025A Survey on Training-free Alignment of Large Language Models
  7. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models
  8. 2026Towards Reliable Truth-Aligned Uncertainty Estimation in Large Language Models