mlsanityData to 14 Oct 2026

DARC: Disagreement-Aware Alignment via Risk-Constrained Decoding

arXiv 2603.08145 ↗First posted 9 Mar 2026cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Preference-based alignment methods (e.g., RLHF, DPO) typically optimize a single scalar objective, implicitly averaging over heterogeneous human preferences. In practice, systematic annotator and user-group disagreement makes mean-reward maximization brittle and susceptible to proxy over-optimization. We propose **Disagreement-Aware Alignment via Risk-Constrained Decoding (DARC)**, a retraining-free inference-time method that frames response selection as distributionally robust, risk-sensitive decision making. Given multiple preference samples or scalable disagreement proxies, DARC reranks candidates by maximizing a *KL-robust (entropic)* satisfaction objective, and provides simple deployment controls that cap or penalize the corresponding entropic risk premium relative to the mean, enabling explicit risk budgets without retraining. We provide theoretical characterization linking this decoding rule to principled pessimism and KL-based distributionally robust optimization. Experiments on alignment benchmarks show that DARC reduces disagreement and tail risk while maintaining competitive average quality under noisy, heterogeneous feedback.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 50 of its 81 references that are in the map. First posted on arXiv 9 Mar 2026.

Builds on

50 of 81 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025Inference-Time Reward Hacking in Large Language Models24 Jun
  2. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignment27 Mar
  3. 2025IPO: Your Language Model is Secretly a Preference Classifier22 Feb
  4. 2025Evaluation of Best-of-N Sampling Strategies for Language Model Alignment18 Feb
  5. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization10 Jul
  6. 2024PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferences12 Jun
  7. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coverage3 Jun
  8. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  9. 2024MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models25 Mar
  10. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  11. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar
  12. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment29 Feb
2 more
  1. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  2. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 36

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  6. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  7. 2017Deep reinforcement learning from human preferencescited by 3,912
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2021Convex optimizationcited by 2,515
  10. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,352
  11. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  12. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
24 more
  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  2. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  3. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  4. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  5. 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 335
  6. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
  7. 2016Statistics of Robust Optimization: A Generalized Empirical Likelihood Approachcited by 183
  8. 2020Large-Scale Methods for Distributionally Robust Optimizationcited by 179
  9. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 176
  10. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 170
  11. 2016Variance-based regularization with convex objectivescited by 155
  12. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 143
  13. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 142
  14. 2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 130
  15. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 123
  16. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  17. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 95
  18. 2015Policy Gradient for Coherent Risk Measurescited by 76
  19. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  20. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignmentcited by 61
  21. 2024Diverging Preferences: When do Annotators Disagree and do Models Know?cited by 47
  22. 2024RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignmentcited by 23
  23. 2024Towards Reliable Alignment: Uncertainty-aware RLHFcited by 7
  24. 2025ENCORE: Entropy-guided Reward Composition for Multi-head Safety Reward Modelscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Conformal Feedback Alignment: Quantifying Answer-Level Reliability for Robust LLM Alignment
  2. 2025Pairwise Calibrated Rewards for Pluralistic Alignment
  3. 2024REFA: Reference Free Alignment for multi-preference optimization
  4. 2025Robust Preference Alignment via Directional Neighborhood Consensus
  5. 2026Reliability-Aware LLM Alignment from Inconsistent Human Feedback
  6. 2026A Statistical Framework for Alignment with Biased AI Feedback
  7. 2025Leveraging Robust Optimization for LLM Alignment under Distribution Shifts
  8. 2025REWARD CONSISTENCY: Improving Multi-Objective Alignment from a Data-Centric Perspective