mlsanityData to 14 Oct 2026

Decision-Making Under Uncertainty: Beyond Probabilities

arXiv 2303.05848 ↗First posted 10 Mar 2023cs.AI cs.LG cs.ROcited by 9

Not on the map. Why?

Abstract (arXiv)

This position paper reflects on the state-of-the-art in decision-making under uncertainty. A classical assumption is that probabilities can sufficiently capture all uncertainty in a system. In this paper, the focus is on the uncertainty that goes beyond this classical interpretation, particularly by employing a clear distinction between aleatoric and epistemic uncertainty. The paper features an overview of Markov decision processes (MDPs) and extensions to account for partial observability and adversarial behavior. These models sufficiently capture aleatoric uncertainty but fail to account for epistemic uncertainty robustly. Consequently, we present a thorough overview of so-called uncertainty models that exhibit uncertainty in a more robust interpretation. We show several solution techniques for both discrete and continuous models, ranging from formal verification, over control-based abstractions, to reinforcement learning. As an integral part of this paper, we list and discuss several key challenges that arise when dealing with rich types of uncertainty in a model-based fashion.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 54 of its 173 references that are in the map. First posted on arXiv 10 Mar 2023.

Builds on

54 of 173 references.

References · 54

  1. 2021Convex optimizationcited by 2,515
  2. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  5. 2017Constrained Policy Optimizationcited by 983
  6. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 913
  7. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 465
  8. 2017Safe Reinforcement Learning via Shieldingcited by 449
  9. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
  10. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  11. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  12. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 164
38 more
  1. 2017A General Safety Framework for Learning-Based Control in Uncertain Robotic Systemscited by 163
  2. 2017Robust Deep Reinforcement Learning with Adversarial Attackscited by 154
  3. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 144
  4. 2022Robust Reinforcement Learning using Offline Datacited by 96
  5. 2020Reinforcement Learning for Non-Stationary Markov Decision Processes: The Blessing of (More) Optimismcited by 76
  6. 2019Estimating Risk and Uncertainty in Deep Reinforcement Learningcited by 70
  7. 2018Efficient Exploration through Bayesian Deep Q-Networkscited by 67
  8. 2021Risk-Averse Offline Reinforcement Learningcited by 65
  9. 2022Constrained Policy Optimization via Bayesian World Modelscited by 57
  10. 2022Safe Reinforcement Learning via Shielding under Partial Observabilitycited by 56
  11. 2016Safe Policy Improvement by Minimizing Robust Baseline Regretcited by 53
  12. 2019Beyond Confidence Regions: Tight Bayesian Ambiguity Sets for Robust MDPscited by 40
  13. 2020Model-based Reinforcement Learning: A Surveycited by 39
  14. 2019PAC Statistical Model Checking for Markov Decision Processes and Stochastic Gamescited by 32
  15. 2021Direct Behavior Specification via Constrained Reinforcement Learningcited by 31
  16. 2018A Sliding-Window Algorithm for Markov Decision Processes with Arbitrarily Changing Rewards and Transitionscited by 30
  17. 2020Robustifying Reinforcement Learning Agents via Action Space Adversarial Trainingcited by 30
  18. 2021Risk-Averse Bayes-Adaptive Reinforcement Learningcited by 29
  19. 2013Solving POMDPs by Searching the Space of Finite Policiescited by 26
  20. 2018Robust Markov Decision Process: Beyond Rectangularitycited by 24
  21. 2020Robust Finite-State Controllers for Uncertain POMDPscited by 24
  22. 2019Safe Policy Improvement with Soft Baseline Bootstrappingcited by 22
  23. 2021Automated Verification and Synthesis of Stochastic Hybrid Systems: A Surveycited by 20
  24. 2021Minimum-Delay Adaptation in Non-Stationary Reinforcement Learning via Online High-Confidence Change-Point Detectioncited by 18
  25. 2019Safe Policy Improvement with an Estimated Baseline Policycited by 17
  26. 2022Robust Anytime Learning of Markov Decision Processescited by 16
  27. 2023Robust Control for Dynamical Systems With Non-Gaussian Noise via Formal Abstractionscited by 16
  28. 2018Bayesian Reinforcement Learning in Factored POMDPscited by 15
  29. 2013What is Decidable about Partially Observable Markov Decision Processes with ω-Regular Objectivescited by 14
  30. 2014Multiple-Environment Markov Decision Processescited by 13
  31. 2019StocHy: automated verification and synthesis of stochastic processescited by 13
  32. 2021Sampling-Based Robust Control of Autonomous Systems with Non-Gaussian Noisecited by 13
  33. 2016POMDP-lite for Robust Robot Planning under Uncertaintycited by 12
  34. 2023Safe Policy Improvement for POMDPs via Finite-State Controllerscited by 12
  35. 2022Probabilities Are Not Enough: Formal Controller Synthesis for Stochastic Dynamical Models with Epistemic Uncertaintycited by 11
  36. 2021Scenario-Based Verification of Uncertain Parametric MDPscited by 8
  37. 2022Constructing MDP Abstractions Using Data with Formal Guaranteescited by 7
  38. 2015Feedback Refinement Relations for the Synthesis of Symbolic Controllerscited by 6

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 2; 2026: 3.

9 papers.

  1. 2024Robust Markov Decision Processes: A Place Where AI and Formal Methods Meetcited by 18
  2. 2024What Are the Odds? Improving the foundations of Statistical Model Checkingcited by 5
  3. 2025Multi-Environment POMDPs: Discrete Model Uncertainty Under Partial Observabilitycited by 3
  4. 2023Efficient Sensitivity Analysis for Parametric Robust Markov Chainscited by 2
  5. 2024Sequential Decision-Making under Uncertainty: A Robust MDPs reviewcited by 2
  6. 2025Diffusion-Based Forecasting for Uncertainty-Aware Model Predictive Controlcited by 2
  7. 2026Robust Shielding for Safe Reinforcement Learningcited by 2
  8. 2026Property-driven Causal Abstractions for Markov Decision Processescited by 0
  9. 2026From Answers to Interpretations: Rethinking Ambiguity-Induced Aleatoric Uncertainty Estimation in LLMscited by 0