mlsanityData to 13 Oct 2026

Non-stationary Risk-sensitive Reinforcement Learning: Near-optimal Dynamic Regret, Adaptive Detection, and Separation Design

arXiv 2211.10815 ↗First posted 19 Nov 2022cs.LG math.OCcited by 4

Not on the map. Why?

Abstract (arXiv)

We study risk-sensitive reinforcement learning (RL) based on an entropic risk measure in episodic non-stationary Markov decision processes (MDPs). Both the reward functions and the state transition kernels are unknown and allowed to vary arbitrarily over time with a budget on their cumulative variations. When this variation budget is known a prior, we propose two restart-based algorithms, namely Restart-RSMB and Restart-RSQ, and establish their dynamic regrets. Based on these results, we further present a meta-algorithm that does not require any prior knowledge of the variation budget and can adaptively detect the non-stationarity on the exponential value functions. A dynamic regret lower bound is then established for non-stationary risk-sensitive RL to certify the near-optimality of the proposed algorithms. Our results also show that the risk control and the handling of the non-stationarity can be separately designed in the algorithm if the variation budget is known a prior, while the non-stationary detection mechanism in the adaptive algorithm depends on the risk parameter. This work offers the first non-asymptotic theoretical analyses for the non-stationary risk-sensitive RL in the literature.

Key facts. Cited by 4 papers in the map (data through 13 Oct 2026). Builds on 15 of its 45 references that are in the map. First posted on arXiv 19 Nov 2022.

Builds on

15 of 45 references.

References · 15

  1. 2018Is Q-learning Provably Efficient?cited by 485
  2. 2014Algorithms for CVaR Optimization in MDPscited by 99
  3. 2014Optimizing the CVaR via Samplingcited by 98
  4. 2021Non-stationary Reinforcement Learning without Prior Knowledge: An Optimal Black-box Approachcited by 98
  5. 2020Reinforcement Learning for Non-Stationary Markov Decision Processes: The Blessing of (More) Optimismcited by 76
  6. 2015Policy Gradient for Coherent Risk Measurescited by 75
  7. 2019A New Algorithm for Non-stationary Contextual Bandits: Efficient, Optimal, and Parameter-freecited by 71
  8. 2020Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regretcited by 58
  9. 2021A Simple Approach for Non-stationary Linear Banditscited by 54
  10. 2021Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 43
  11. 2020A Kernel-Based Approach to Non-Stationary Reinforcement Learning in Metric Spacescited by 32
  12. 2020Efficient Learning in Non-Stationary Linear Markov Decision Processescited by 26
3 more
  1. 2020Nonstationary Reinforcement Learning with Linear Function Approximationcited by 23
  2. 2020Model-Free Non-Stationary RL: Near-Optimal Regret and Applications in Multi-Agent RL and Inventory Controlcited by 16
  3. 2021Optimistic Policy Optimization is Provably Efficient in Non-stationary MDPscited by 16

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 1.

4 papers.

  1. 2023Regret Bounds for Markov Decision Processes with Recursive Optimized Certainty Equivalentscited by 15
  2. 2023Regret Bounds for Risk-sensitive Reinforcement Learning with Lipschitz Dynamic Risk Measurescited by 8
  3. 2023Tempo Adaptation in Non-stationary Reinforcement Learningcited by 3
  4. 2024Pausing Policy Learning in Non-stationary Reinforcement Learningcited by 2