mlsanityData to 7 Oct 2026

Provably Efficient Partially Observable Risk-Sensitive Reinforcement Learning with Hindsight Observation

arXiv 2402.18149 ↗First posted 28 Feb 2024cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

This work pioneers regret analysis of risk-sensitive reinforcement learning in partially observable environments with hindsight observation, addressing a gap in theoretical exploration. We introduce a novel formulation that integrates hindsight observations into a Partially Observable Markov Decision Process (POMDP) framework, where the goal is to optimize accumulated reward under the entropic risk measure. We develop the first provably efficient RL algorithm tailored for this setting. We also prove by rigorous analysis that our algorithm achieves polynomial regret tildeOleft(frace^|γ|H-1|γ|HH2√KHS2OAright), which outperforms or matches existing upper bounds when the model degenerates to risk-neutral or fully observable settings. We adopt the method of change-of-measure and develop a novel analytical tool of beta vectors to streamline mathematical derivations. These techniques are of particular interest to the theoretical study of reinforcement learning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 23 of its 59 references that are in the map. First posted on arXiv 28 Feb 2024.

Builds on

23 of 59 references.

References · 23

  1. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
  2. 2019Learning by Cheatingcited by 361
  3. 2017Asymmetric Actor Critic for Image-Based Robot Learningcited by 202
  4. 2020Provable Self-Play Algorithms for Competitive Reinforcement Learningcited by 112
  5. 2022When Is Partially Observable Reinforcement Learning Not Scary?cited by 100
  6. 2020Sample-Efficient Reinforcement Learning of Undercomplete POMDPscited by 66
  7. 2020Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regretcited by 57
  8. 2021Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 42
  9. 2021RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agentscited by 31
  10. 2021Deep Hedging of Derivatives Using Reinforcement Learningcited by 30
  11. 2022Learning in Observable POMDPs, without Computationally Intractable Oraclescited by 30
  12. 2022Planning in Observable POMDPs in Quasipolynomial Timecited by 25
11 more
  1. 2022Hindsight Learning for MDPs with Exogenous Inputscited by 25
  2. 2022Provably Efficient Risk-Sensitive Reinforcement Learning: Iterated CVaR and Worst Pathcited by 21
  3. 2023RiskQ: Risk-sensitive Multi-Agent Reinforcement Learning Value Factorizationcited by 19
  4. 2022Optimistic PAC Reinforcement Learning: the Instance-Dependent Viewcited by 17
  5. 2023On Dynamic Programming Decompositions of Static Risk Measures in Markov Decision Processescited by 17
  6. 2022Reinforcement Learning from Partial Observation: Linear Function Approximation with Provable Sample Efficiencycited by 15
  7. 2021Risk-Conditioned Distributional Soft Actor-Critic for Risk-Sensitive Navigationcited by 13
  8. 2019Online Risk-Bounded Motion Planning for Autonomous Vehicles in Dynamic Environmentscited by 9
  9. 2022Deep Reinforcement Learning Approach for Trading Automation in The Stock Marketcited by 9
  10. 2023Regret Bounds for Risk-sensitive Reinforcement Learning with Lipschitz Dynamic Risk Measurescited by 7
  11. 2023Sample-Efficient Learning of POMDPs with Multiple Observations In Hindsightcited by 3

Built on by

Not cited yet.