mlsanityData to 7 Oct 2026

Diffusion Policies for Risk-Averse Behavior Modeling in Offline Reinforcement Learning

arXiv 2403.17646 ↗First posted 26 Mar 2024cs.LGcited by 2

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) presents distinct challenges as it relies solely on observational data. A central concern in this context is ensuring the safety of the learned policy by quantifying uncertainties associated with various actions and environmental stochasticity. Traditional approaches primarily emphasize mitigating epistemic uncertainty by learning risk-averse policies, often overlooking environmental stochasticity. In this study, we propose an uncertainty-aware distributional offline RL method to simultaneously address both epistemic uncertainty and environmental stochasticity. We propose a model-free offline RL algorithm capable of learning risk-averse policies and characterizing the entire distribution of discounted cumulative rewards, as opposed to merely maximizing the expected value of accumulated discounted returns. Our method is rigorously evaluated through comprehensive experiments in both risk-sensitive and risk-neutral benchmarks, demonstrating its superior performance.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 26 of its 32 references that are in the map. First posted on arXiv 26 Mar 2024.

Builds on

26 of 32 references.

Closest ancestors · 8

Same direction, newest first

  1. 2022One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning30 Nov
  2. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  3. 2021Conservative Offline Distributional Reinforcement Learning12 Jul
  4. 2021Risk-Averse Offline Reinforcement Learning10 Feb
  5. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  8. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 18

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  3. 2022Classifier-Free Diffusion Guidancecited by 5,717
  4. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  7. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  9. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  10. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  11. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  12. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
6 more
  1. 2023Imitating Human Behaviour with Diffusion Modelscited by 260
  2. 2021On Fast Sampling of Diffusion Probabilistic Modelscited by 170
  3. 2019Fully Parameterized Quantile Function for Distributional Reinforcement Learningcited by 87
  4. 2019Worst Cases Policy Gradientscited by 52
  5. 2020Improving Robustness via Risk Averse Distributional Reinforcement Learningcited by 28
  6. 2023Risk-Averse Model Uncertainty for Distributionally Robust Safe Reinforcement Learningcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025RAMAC: Multimodal Risk-Aware Offline Reinforcement Learning and the Role of Behavior Regularizationcited by 0
  2. 2026RS-Diffuser: Risk-Sensitive Diffusion Planning with Distributional Value Guidancecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Efficient and Uncertainty-Aware Diffusion Framework for Offline-to-Online Reinforcement Learning
  2. 2023Selective Uncertainty Propagation in Offline RL
  3. 2020Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluation
  4. 2020Reliable Off-policy Evaluation for Reinforcement Learning
  5. 2025Evaluation-Time Policy Switching for Offline Reinforcement Learning
  6. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning
  7. 2025Towards Optimal Offline Reinforcement Learning
  8. 2024Mitigating Distribution Shift in Model-based Offline RL via Shifts-aware Reward Learning