mlsanityData to 7 Oct 2026

One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning

arXiv 2212.00124 ↗First posted 30 Nov 2022cs.LGcited by 21

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) is suitable for safety-critical domains where online exploration is too costly or dangerous. In such safety-critical settings, decision-making should take into consideration the risk of catastrophic outcomes. In other words, decision-making should be risk-sensitive. Previous works on risk in offline RL combine together offline RL techniques, to avoid distributional shift, with risk-sensitive RL algorithms, to achieve risk-sensitivity. In this work, we propose risk-sensitivity as a mechanism to jointly address both of these issues. Our model-based approach is risk-averse to both epistemic and aleatoric uncertainty. Risk-aversion to epistemic uncertainty prevents distributional shift, as areas not covered by the dataset have high epistemic uncertainty. Risk-aversion to aleatoric uncertainty discourages actions that may result in poor outcomes due to environment stochasticity. Our experiments show that our algorithm achieves competitive performance on deterministic benchmarks, and outperforms existing approaches for risk-sensitive objectives in stochastic domains.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 59 of its 93 references that are in the map. First posted on arXiv 30 Nov 2022.

Builds on

59 of 93 references.

Closest ancestors · 23

Same direction, newest first

  1. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief13 Oct
  2. 2022Offline RL Policies Should be Trained to be Adaptive5 Jul
  3. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations9 Jun
  4. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
  5. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  6. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  7. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  8. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning8 Oct
  9. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  10. 2021Offline Reinforcement Learning with Reverse Model-based Imagination1 Oct
  11. 2021Conservative Offline Distributional Reinforcement Learning12 Jul
  12. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
11 more
  1. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  2. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  3. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  4. 2021Risk-Averse Offline Reinforcement Learning10 Feb
  5. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learning12 Aug
  6. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  7. 2020MOPO: Model-based Offline Policy Optimization27 May
  8. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  10. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  11. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 36

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2020Mastering Atari with Discrete World Modelscited by 990
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  7. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  8. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  9. 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
  10. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  11. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  12. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
24 more
  1. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  2. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  3. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 162
  4. 2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 136
  5. 2020Model-Based Offline Planningcited by 126
  6. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  7. 2014Optimizing the CVaR via Samplingcited by 98
  8. 2015Policy Gradient for Coherent Risk Measurescited by 74
  9. 2020A Game Theoretic Framework for Model Based Reinforcement Learningcited by 72
  10. 2019Estimating Risk and Uncertainty in Deep Reinforcement Learningcited by 68
  11. 2019Being Optimistic to Be Conservative: Quickly Learning a CVaR Policycited by 63
  12. 2017How Should a Robot Assess Risk? Towards an Axiomatic Theory of Risk in Roboticscited by 57
  13. 2019Learning When-to-Treat Policiescited by 49
  14. 2020Cautious Adaptation For Reinforcement Learning in Safety-Critical Settingscited by 42
  15. 2021Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environmentcited by 41
  16. 2021Risk-Averse Bayes-Adaptive Reinforcement Learningcited by 28
  17. 2019Epistemic Risk-Sensitive Reinforcement Learningcited by 17
  18. 2022Real World Offline Reinforcement Learning with Realistic Data Sourcecited by 16
  19. 2019Multi-Agent Deep Reinforcement Learning for Liquidation Strategy Analysiscited by 15
  20. 2021SENTINEL: Taming Uncertainty with Ensemble-based Distributional Reinforcement Learningcited by 15
  21. 2019Robust and Adaptive Planning under Model Uncertaintycited by 10
  22. 2021Two steps to risk sensitivitycited by 8
  23. 2021Planning for Risk-Aversion and Expected Value in MDPscited by 5
  24. 2021Risk Sensitive Model-Based Reinforcement Learning using Uncertainty Guided Planningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 7; 2026: 9.

21 papers.

Sort:

By area:

  1. 2024A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalentscited by 15
  2. 2024Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learningcited by 5
  3. 2025Scalable Decision-Making in Stochastic Environments through Learned Temporal Abstractioncited by 4
  4. 2025Hybrid Cross-domain Robust Reinforcement Learningcited by 3
  5. 2025Recursive Entropic Risk Optimization in Discounted MDPs: Sample Complexity Bounds with a Generative Modelcited by 3
  6. 2023Is Risk-Sensitive Reinforcement Learning Properly Resolved?cited by 2
  7. 2024Diffusion Policies for Risk-Averse Behavior Modeling in Offline Reinforcement Learningcited by 2
  8. 2025Risk-sensitive Actor-Critic with Static Spectral Risk Measures for Online and Offline Reinforcement Learningcited by 2
  9. 2026Actor-Critic Algorithm for Dynamic Expectile and CVaRcited by 2
  10. 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatismcited by 1
  11. 2026Offline Policy Optimization with Posterior Samplingcited by 1
  12. 2023Risk-Sensitive and Robust Model-Based Reinforcement Learning and Planningcited by 0
9 more
  1. 2025Ensemble RL through Classifier Models: Enhancing Risk-Return Trade-offs in Trading Strategiescited by 0
  2. 2025RAMAC: Multimodal Risk-Aware Offline Reinforcement Learning and the Role of Behavior Regularizationcited by 0
  3. 2026In-Context Planning with Latent Temporal Abstractionscited by 0
  4. 2026Action-Conditioned Risk Gating for Safety-Critical Control under Partial Observabilitycited by 0
  5. 2026Pessimistic Risk-Aware Policy Learning in Contextual Banditscited by 0
  6. 2026On the Sample Complexity of Discounted Reinforcement Learning with Optimized Certainty Equivalentscited by 0
  7. 2026Regularized Offline Policy Optimization with Posterior Hybrid Bayesian Beliefcited by 0
  8. 2026Learning from the Test: Self-Referential Differential Testing for Deep RL Agentscited by 0
  9. 2026From Static Policies to Adaptive Priors in Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Bayesian Design Principles for Offline-to-Online Reinforcement Learning
  2. 2023Selective Uncertainty Propagation in Offline RL
  3. 2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learning
  4. 2026Escaping Offline Pessimism: Vector-Field Reward Shaping for Safe Frontier Exploration
  5. 2020Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluation
  6. 2021An Offline Risk-aware Policy Selection Method for Bayesian Markov Decision Processes
  7. 2025Online Optimization for Offline Safe Reinforcement Learning
  8. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity