mlsanityData to 14 Oct 2026

Robust Risk-Sensitive Reinforcement Learning from Corrupted Human Feedback

arXiv 2609.38938 ↗First posted 30 Sep 2026cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Reinforcement learning with human feedback (RLHF) learns from human comparisons, which can be corrupted or deliberately manipulated. This paper studies online risk-sensitive RLHF with static conditional value-at-risk (CVaR) under adversarial preference-label flips. We consider additive linear rewards and a fixed-reference protocol with one comparison per episode and at most C flipped labels over K episodes. We propose weighted streamed-preference CVaR RLHF (WSP-CVaR-RLHF), which combines uncertainty-weighted reward estimation with optimistic augmented-state CVaR planning. For known transitions and normalized rewards, we establish the regret bound widetildeOleft(fracdkappa√fracKα+fracdCkappaαright) up to lower-order terms, where d is the reward-feature dimension, α is the CVaR level, and kappa characterizes the preference link. The bound separates the clean statistical cost from the penalty caused by corrupted feedback. We further extend the analysis to unknown tabular transitions, where the trajectory distribution entering the CVaR objective must be learned together with the reward. We address the resulting coupled uncertainty using rectangular transition confidence sets, joint optimistic planning, and a history-level CVaR simulation argument. Experiments under four adversarial attacks demonstrate that WSP-CVaR-RLHF consistently reduces cumulative regret relative to its unweighted robust counterpart while preserving confidence-set coverage.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 48 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

31 of 48 references.

Closest ancestors · 14

Same direction, newest first

  1. 2026Regret Bounds for Reinforcement Learning from Multi-Source Imperfect Preferences20 Mar
  2. 2026Safe RLHF Beyond Expectation: Stochastic Dominance for Universal Spectral Risk Control11 Mar
  3. 2025Best Policy Learning from Trajectory Preference Feedback31 Jan
  4. 2024RA-PbRL: Provably Efficient Risk-Aware Preference-Based Reinforcement Learning31 Oct
  5. 2024Robust Reinforcement Learning from Corrupted Human Feedback21 Jun
  6. 2024Learning from Imperfect Human Feedback: a Tale from Corruption-Robust Dueling18 May
  7. 2024Nearly Optimal Algorithms for Contextual Dueling Bandits from Adversarial Feedback16 Apr
  8. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Model11 Feb
  9. 2024Corruption Robust Offline Reinforcement Learning with Human Feedback9 Feb
  10. 2023Provably Efficient Iterated CVaR Reinforcement Learning with Function Approximation and Human Feedback6 Jul
  11. 2023Is RLHF More Difficult than Standard RL?25 Jun
  12. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
2 more
  1. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  2. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 17

  1. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 170
  2. 2014Optimizing the CVaR via Samplingcited by 99
  3. 2014Algorithms for CVaR Optimization in MDPscited by 99
  4. 2018Batch Active Preference-Based Learning of Reward Functionscited by 87
  5. 2015Policy Gradient for Coherent Risk Measurescited by 76
  6. 2019Corruption-robust exploration in episodic reinforcement learningcited by 72
  7. 2021A Model Selection Approach for Corruption Robust Reinforcement Learningcited by 45
  8. 2023Near-Minimax-Optimal Risk-Sensitive Reinforcement Learning with CVaRcited by 32
  9. 2022Provably Efficient Risk-Sensitive Reinforcement Learning: Iterated CVaR and Worst Pathcited by 22
  10. 2014Policy Gradients for CVaR-Constrained MDPscited by 21
  11. 2025When Human Preferences Flip: An Instance-Dependent Robust Loss for RLHFcited by 3
  12. 2024Robust Risk-Sensitive Reinforcement Learning with Conditional Value-at-Riskcited by 2
5 more
  1. 2025Risk-aware Direct Preference Optimization under Nested Risk Measurecited by 2
  2. 2025Reinforcement Learning from Adversarial Preferences in Tabular MDPscited by 2
  3. 2026Structure from Strategic Interaction & Uncertainty: Risk Sensitive Games for Robust Preference Learningcited by 2
  4. 2026Adversarially Robust Control of Conditional Value-at-Risk via Rockafellar-Uryasev Conformal Inferencecited by 2
  5. 2025Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignmentcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Wasserstein Distributionally Robust Regret Optimization for Reinforcement Learning from Human Feedback
  2. 2026A Unifying Lens on Reward Uncertainty in RLHF
  3. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  4. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF
  5. 2025Towards Efficient Online Exploration for Reinforcement Learning with Human Feedback
  6. 2023Contrastive Preference Learning: Learning from Human Feedback without RL
  7. 2026Policy Gradient Primal-Dual Method for Safe Reinforcement Learning from Human Feedback
  8. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback