mlsanityData to 7 Oct 2026

RA-PbRL: Provably Efficient Risk-Aware Preference-Based Reinforcement Learning

arXiv 2410.23569 ↗First posted 31 Oct 2024cs.LGcited by 8

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) has recently surged in popularity, particularly for aligning large language models and other AI systems with human intentions. At its core, RLHF can be viewed as a specialized instance of Preference-based Reinforcement Learning (PbRL), where the preferences specifically originate from human judgments rather than arbitrary evaluators. Despite this connection, most existing approaches in both RLHF and PbRL primarily focus on optimizing a mean reward objective, neglecting scenarios that necessitate risk-awareness, such as AI safety, healthcare, and autonomous driving. These scenarios often operate under a one-episode-reward setting, which makes conventional risk-sensitive objectives inapplicable. To address this, we explore and prove the applicability of two risk-aware objectives to PbRL : nested and static quantile risk objectives. We also introduce Risk-AwarePbRL (RA-PbRL), an algorithm designed to optimize both nested and static objectives. Additionally, we provide a theoretical analysis of the regret upper bounds, demonstrating that they are sublinear with respect to the number of episodes, and present empirical results to support our findings. Our code is available in https://github.com/aguilarjose11/PbRLNeurips.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 19 of its 36 references that are in the map. First posted on arXiv 31 Oct 2024.

Builds on

19 of 36 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Making RL with Preference-based Feedback Efficient via Randomization23 Oct
  2. 2023Provably Efficient Iterated CVaR Reinforcement Learning with Function Approximation and Human Feedback6 Jul
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 16

  1. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  2. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  3. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 233
  4. 2019Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Boundscited by 194
  5. 2017Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learningcited by 175
  6. 2013Risk-sensitive Reinforcement Learningcited by 129
  7. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
  8. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 80
  9. 2020Preference-based Reinforcement Learning with Finite-Time Guaranteescited by 65
  10. 2020Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regretcited by 57
  11. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  12. 2013Learning Trajectory Preferences for Manipulators via Iterative Improvementcited by 31
4 more
  1. 2022Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 26
  2. 2019Epistemic Risk-Sensitive Reinforcement Learningcited by 17
  3. 2016A First Look at Ad-block Detection: A New Arms Race on the Webcited by 15
  4. 2023Risk-sensitive Markov Decision Process and Learning under General Utility Functionscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 3.

8 papers.

  1. 2025Modeling and Optimizing User Preferences in AI Copilots: A Comprehensive Survey and Taxonomycited by 3
  2. 2025Recursive Entropic Risk Optimization in Discounted MDPs: Sample Complexity Bounds with a Generative Modelcited by 3
  3. 2025TW-CRL: Time-Weighted Contrastive Reward Learning for Efficient Inverse Reinforcement Learningcited by 2
  4. 2025Risk-aware Direct Preference Optimization under Nested Risk Measurecited by 2
  5. 2025Constrained Language Model Policy Optimization via Risk-aware Stepwise Alignmentcited by 0
  6. 2026On the Sample Complexity of Discounted Reinforcement Learning with Optimized Certainty Equivalentscited by 0
  7. 2026Generalizing Preference-based Reinforcement Learning: a Rationality Model for Incomparabilitycited by 0
  8. 2026Robust Risk-Sensitive Reinforcement Learning from Corrupted Human Feedbackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learning
  2. 2023Direct Preference-based Policy Optimization without Reward Modeling
  3. 2025Offline Safe Policy Optimization From Heterogeneous Feedback
  4. 2023Is RLHF More Difficult than Standard RL?
  5. 2023A Survey of Reinforcement Learning from Human Feedback
  6. 2025Policy-labeled Preference Learning: Is Preference Enough for RLHF?
  7. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  8. 2025One Framework to Rule Them All: Unifying RL-Based and RL-Free Methods in RLHF