Risk-aware Direct Preference Optimization under Nested Risk Measure
On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods
Abstract (arXiv)
When fine-tuning pre-trained Large Language Models (LLMs) to align with human values and intentions, maximizing the estimated reward can lead to superior performance, but it also introduces potential risks due to deviations from the reference model's intended behavior. Most existing methods typically introduce KL divergence to constrain deviations between the trained model and the reference model; however, this may not be sufficient in certain applications that require tight risk control. In this paper, we introduce Risk-aware Direct Preference Optimization (Ra-DPO), a novel approach that incorporates risk-awareness by employing a class of nested risk measures. This approach formulates a constrained risk-aware advantage function maximization problem and then converts the Bradley-Terry model into a token-level representation. The objective function maximizes the likelihood of the policy while suppressing the deviation between a trained model and the reference model using a sequential risk ratio, thereby enhancing the model's risk-awareness. Experimental results across three open-source datasets: IMDb Dataset, Anthropic HH Dataset, and AlpacaEval, demonstrate the proposed method's superior performance in balancing alignment performance and model drift. Our code is opensourced at https://github.com/zlj123-max/Ra-DPO.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 37 of its 63 references that are in the map. First posted on arXiv 26 May 2025.
Builds on
37 of 63 references.
Closest ancestors · 9
Same direction, newest first
- 2025Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model22 Apr
- 2024Robust Preference Optimization through Reward Model Distillation29 May
- 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
- 2024Token-level Direct Preference Optimization18 Apr
- 2024Stepwise Alignment for Constrained Language Model Policy Optimization17 Apr
- 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
- 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
- 2023Preference Ranking Optimization for Human Alignment30 Jun
- 2022Training language models to follow instructions with human feedback4 Mar
Other references · 28
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
- 2017Deep reinforcement learning from human preferencescited by 3,912
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
- 2015Trust Region Policy Optimizationcited by 2,580
- 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
- 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
- 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
- 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
16 more
- 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
- 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 223
- 2018An Introduction to Deep Reinforcement Learningcited by 206
- 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181
- 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 170
- 2024Group Robust Preference Optimization in Reward-free RLHFcited by 95
- 2020Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regretcited by 58
- 2023Near-Minimax-Optimal Risk-Sensitive Reinforcement Learning with CVaRcited by 32
- 2022Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 26
- 2022Provably Efficient Risk-Sensitive Reinforcement Learning: Iterated CVaR and Worst Pathcited by 22
- 2021Robust Risk-Aware Reinforcement Learningcited by 18
- 2023Provably Efficient Iterated CVaR Reinforcement Learning with Function Approximation and Human Feedbackcited by 14
- 2021Risk-averse autonomous systems: A brief history and recent developments from the perspective of optimal controlcited by 12
- 2024RA-PbRL: Provably Efficient Risk-Aware Preference-Based Reinforcement Learningcited by 9
- 2024Token-level Proximal Policy Optimization for Query Generationcited by 7
- 2025Risk-Averse Finetuning of Large Language Modelscited by 7
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
- 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
- 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
- 2026Improving Safety Alignment via Balanced Direct Preference Optimization
- 2024Multi-Reference Preference Optimization for Large Language Models
- 2025Preference Optimization by Estimating the Ratio of the Data Distribution
- 2025KL Penalty Control via Perturbation for Direct Preference Optimization
- 2025SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety