mlsanityData to 14 Oct 2026

Distributional Model Equivalence for Risk-Sensitive Reinforcement Learning

arXiv 2307.01708 ↗First posted 4 Jul 2023cs.LG cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

We consider the problem of learning models for risk-sensitive reinforcement learning. We theoretically demonstrate that proper value equivalence, a method of learning models which can be used to plan optimally in the risk-neutral setting, is not sufficient to plan optimally in the risk-sensitive setting. We leverage distributional reinforcement learning to introduce two new notions of model equivalence, one which is general and can be used to plan for any risk measure, but is intractable; and a practical variation which allows one to choose which risk measures they may plan optimally for. We demonstrate how our framework can be used to augment any model-free risk-sensitive algorithm, and provide both tabular and large-scale experiments to demonstrate its ability.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 18 of its 51 references that are in the map. First posted on arXiv 4 Jul 2023.

Builds on

18 of 51 references.

References · 18

  1. 2016OpenAI Gymcited by 1,736
  2. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  3. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  4. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  5. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  6. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 172
  7. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 170
  8. 2014Algorithms for CVaR Optimization in MDPscited by 99
  9. 2020The Value Equivalence Principle for Model-Based Reinforcement Learningcited by 94
  10. 2015Policy Gradient for Coherent Risk Measurescited by 76
  11. 2018An Analysis of Categorical Distributional Reinforcement Learningcited by 68
  12. 2019Statistics and Samples in Distributional Reinforcement Learningcited by 63
6 more
  1. 2022Efficient Risk-Averse Reinforcement Learningcited by 43
  2. 2021Proper Value Equivalencecited by 33
  3. 2022Value Gradient weighted Model-Based Reinforcement Learningcited by 32
  4. 2020Policy-Aware Model Learning for Policy Gradient Methodscited by 23
  5. 2019Gradient-Aware Model-based Policy Searchcited by 19
  6. 2022Deciding What to Model: Value-Equivalent Sampling for Reinforcement Learningcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 3; 2026: 1.

9 papers.

  1. 2024Foundations of Multivariate Distributional Reinforcement Learningcited by 17
  2. 2024MAD-TD: Model-Augmented Data stabilizes High Update Ratio RLcited by 16
  3. 2024Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learningcited by 7
  4. 2025A Finite Sample Analysis of Distributional TD Learning with Linear Function Approximationcited by 6
  5. 2023Value-Distributional Model-Based Reinforcement Learningcited by 5
  6. 2024Risk-averse Total-reward MDPs with ERM and EVaRcited by 4
  7. 2025Accelerated Distributional Temporal Difference Learning with Linear Function Approximationcited by 1
  8. 2025Risk-Averse Total-Reward Reinforcement Learningcited by 0
  9. 2026Risk-Aware Human-in-the-Loop Framework with Adaptive Intrusion Response for Autonomous Vehiclescited by 0