Online Policy Evaluation for MDPs with Dynamic UBSR Measures
On the map: Deep reinforcement learning methods → RL theory and sample complexity
Abstract (arXiv)
Developing efficient function-approximation methods for policy evaluation is a fundamental challenge in risk-aware reinforcement learning. Existing approaches either focus on restrictive classes of risk measures or rely on access to a simulator, limiting their applicability in fully online settings. In this work, we propose computationally efficient online learning algorithms for policy evaluation in Markov decision processes (MDPs) with dynamic utility-based shortfall risk (UBSR) measures under linear function approximation. Specifically, we introduce the UBSR-TD algorithm, establish conditions under which it converges almost surely, and develop several variants designed to accelerate convergence. Our formulation shows that existing policy evaluation algorithms for risk-neutral MDPs can be readily adapted to dynamic UBSR settings by incorporating a loss function into the temporal-difference error. Numerical experiments support our theoretical findings, and an application to a perishable inventory management problem with shelf-life uncertainty demonstrates the practical effectiveness of the proposed methods.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 20 of its 47 references that are in the map. First posted on arXiv 25 Jul 2026.
Builds on
20 of 47 references.
Closest ancestors · 4
Same direction, newest first
- 2026Actor-Critic Algorithm for Dynamic Expectile and CVaR8 May
- 2025Planning and Learning in Average Risk-aware MDPs22 Mar
- 2024Q-learning for Quantile MDPs: A Decomposition, Performance, and Convergence Analysis31 Oct
- 2024The ODE Method for Stochastic Approximation and Reinforcement Learning with Markovian Noise15 Jan
Other references · 16
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2013Risk-sensitive Reinforcement Learningcited by 130
- 2015Policy Gradient for Coherent Risk Measurescited by 76
- 2020Logistic Q-Learningcited by 34
- 2021The ODE Method for Asymptotic Statistics in Stochastic Approximation and Reinforcement Learningcited by 32
- 2021Reinforcement Learning with Dynamic Convex Risk Measurescited by 21
- 2021On the Convergence and Optimality of Policy Gradient for Markov Coherent Riskcited by 20
- 2023Soft Robust MDPs and Risk-Sensitive MDPs: Equivalence, Policy Gradient, and Sample Complexitycited by 17
- 2020Risk-Averse Learning by Temporal Difference Methodscited by 13
- 2021Risk-averse autonomous systems: A brief history and recent developments from the perspective of optimal controlcited by 12
- 2016Accelerated Gradient Temporal Difference Learningcited by 10
- 2023A Concentration Bound for TD(0) with Function Approximationcited by 10
4 more
- 2017Approximate Value Iteration for Risk-aware Markov Decision Processescited by 9
- 2022Conditionally Elicitable Dynamic Risk Measures for Deep Reinforcement Learningcited by 9
- 2023Online Estimation and Inference for Robust Policy Evaluation in Reinforcement Learningcited by 4
- 2023Platelet Inventory Management with Approximate Dynamic Programmingcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Bayesian Risk-Sensitive Policy Optimization For MDPs With General Loss Functions
- 2024Risk-averse Total-reward MDPs with ERM and EVaR
- 2026Long-Run Conditional Value-at-Risk Reinforcement Learning
- 2024On Policy Evaluation Algorithms in Distributional Reinforcement Learning
- 2025Risk-Averse Total-Reward Reinforcement Learning
- 2026Risk-Aware General-Utility Markov Decision Processes
- 2024Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning
- 2026Towards Optimal Policy Improvement