mlsanityData to 7 Oct 2026

Provably Efficient Iterated CVaR Reinforcement Learning with Function Approximation and Human Feedback

arXiv 2307.02842 ↗First posted 6 Jul 2023cs.LGcited by 13

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Risk-sensitive reinforcement learning (RL) aims to optimize policies that balance the expected reward and risk. In this paper, we present a novel risk-sensitive RL framework that employs an Iterated Conditional Value-at-Risk (CVaR) objective under both linear and general function approximations, enriched by human feedback. These new formulations provide a principled way to guarantee safety in each decision making step throughout the control process. Moreover, integrating human feedback into risk-sensitive RL framework bridges the gap between algorithmic decision-making and human participation, allowing us to also guarantee safety for human-in-the-loop systems. We propose provably sample-efficient algorithms for this Iterated CVaR RL and provide rigorous theoretical analysis. Furthermore, we establish a matching lower bound to corroborate the optimality of our algorithms in a linear context.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 68 references that are in the map. First posted on arXiv 6 Jul 2023.

Builds on

32 of 68 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Is RLHF More Difficult than Standard RL?25 Jun
  2. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  3. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  4. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 28

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  4. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  5. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  6. 2013Learning to Optimize Via Posterior Samplingcited by 253
  7. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 233
  8. 2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 224
  9. 2019Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Boundcited by 199
  10. 2020Nearly Minimax Optimal Reinforcement Learning for Linear Mixture Markov Decision Processescited by 199
  11. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 162
  12. 2014Model-based Reinforcement Learning and the Eluder Dimensioncited by 116
16 more
  1. 2020Provably Efficient Reinforcement Learning for Discounted MDPs with Feature Mappingcited by 102
  2. 2019Sample Complexity of Reinforcement Learning using Linearly Combined Model Ensemblescited by 100
  3. 2022When Is Partially Observable Reinforcement Learning Not Scary?cited by 100
  4. 2022Nearly Minimax Optimal Reinforcement Learning for Linear Markov Decision Processescited by 57
  5. 2019Safe Reinforcement Learning on Autonomous Vehiclescited by 48
  6. 2021Improved Variance-Aware Confidence Sets for Linear Bandits and Linear Mixture MDPcited by 45
  7. 2022VOQL: Towards Optimal Regret in Model-free RL with Nonlinear Function Approximationcited by 44
  8. 2023Variance-Dependent Regret Bounds for Linear Bandits and Reinforcement Learning: Adaptivity and Computational Efficiencycited by 39
  9. 2020Safe Reinforcement Learning for Autonomous Vehicles through Parallel Constrained Policy Optimizationcited by 34
  10. 2023Near-Minimax-Optimal Risk-Sensitive Reinforcement Learning with CVaRcited by 32
  11. 2022Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 26
  12. 2022Provably Efficient Risk-Sensitive Reinforcement Learning: Iterated CVaR and Worst Pathcited by 21
  13. 2021Understanding the Eluder Dimensioncited by 19
  14. 2023Regret Bounds for Markov Decision Processes with Recursive Optimized Certainty Equivalentscited by 14
  15. 2023Safety Analysis in the Era of Large Language Models: A Case Study of STPA using ChatGPTcited by 13
  16. 2017Approximate Value Iteration for Risk-aware Markov Decision Processescited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 5; 2026: 3.

13 papers.

Sort:

By area:

  1. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 15
  2. 2024RA-PbRL: Provably Efficient Risk-Aware Preference-Based Reinforcement Learningcited by 8
  3. 2023Provably Efficient CVaR RL in Low-rank MDPscited by 6
  4. 2024Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learningcited by 5
  5. 2025Near-Optimal Sample Complexity for Iterated CVaR Reinforcement Learning with a Generative Modelcited by 4
  6. 2024Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximationcited by 3
  7. 2026AI Alignment Breaks at the Edgecited by 3
  8. 2025Adversarial Policy Optimization for Offline Preference-based Reinforcement Learningcited by 2
  9. 2025Risk-aware Direct Preference Optimization under Nested Risk Measurecited by 2
  10. 2025Reducing the Probability of Undesirable Outputs in Language Models Using Probabilistic Inferencecited by 0
  11. 2025Constrained Language Model Policy Optimization via Risk-aware Stepwise Alignmentcited by 0
  12. 2026On the Sample Complexity of Discounted Reinforcement Learning with Optimized Certainty Equivalentscited by 0
1 more
  1. 2026Robust Risk-Sensitive Reinforcement Learning from Corrupted Human Feedbackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Risk-sensitive Actor-Critic with Static Spectral Risk Measures for Online and Offline Reinforcement Learning
  2. 2026Safe RLHF Beyond Expectation: Stochastic Dominance for Universal Spectral Risk Control
  3. 2022One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning
  4. 2022On Practical Reinforcement Learning: Provable Robustness, Scalability, and Statistical Efficiency
  5. 2019Active Learning for Risk-Sensitive Inverse Reinforcement Learning
  6. 2026Policy Gradient Primal-Dual Method for Safe Reinforcement Learning from Human Feedback
  7. 2025Reinforcement Learning from Human Feedback
  8. 2024Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization