# Certifiably Robust Policies for Uncertain Parametric Environments

arXiv: https://arxiv.org/abs/2408.03093 · HTML: https://mlsanity.com/paper/2408.03093/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 58 references that are in the map.
- First posted on arXiv 6 Aug 2024.

## Builds on (most cited first, 18 of 18)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Distral: Robust Multitask Reinforcement Learning](https://mlsanity.com/paper/1707.04175/) (2017)
- [Meta-Reinforcement Learning of Structured Exploration Strategies](https://mlsanity.com/paper/1802.07245/) (2018)
- [Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability](https://mlsanity.com/paper/2107.06277/) (2021)
- [A Tutorial on Meta-Reinforcement Learning](https://mlsanity.com/paper/2301.08028/) (2023)
- [Online Robust Reinforcement Learning with Model Uncertainty](https://mlsanity.com/paper/2109.14523/) (2021)
- [A Bayesian Approach to Robust Reinforcement Learning](https://mlsanity.com/paper/1905.08188/) (2019)
- [Safety-Constrained Reinforcement Learning for MDPs](https://mlsanity.com/paper/1510.05880/) (2015)
- [PAC Statistical Model Checking for Markov Decision Processes and Stochastic Games](https://mlsanity.com/paper/1905.04403/) (2019)
- [Robust Anytime Learning of Markov Decision Processes](https://mlsanity.com/paper/2205.15827/) (2022)
- [Multiple-Environment Markov Decision Processes](https://mlsanity.com/paper/1405.4733/) (2014)
- [Train Hard, Fight Easy: Robust Meta Reinforcement Learning](https://mlsanity.com/paper/2301.11147/) (2023)
- [Faster Statistical Model Checking for Unbounded Temporal Properties](https://mlsanity.com/paper/1504.05739/) (2015)
- [Scenario-Based Verification of Uncertain Parametric MDPs](https://mlsanity.com/paper/2112.13020/) (2021)
- [Sampling-Based Verification of CTMCs with Uncertain Rates](https://mlsanity.com/paper/2205.08300/) (2022)
- [What Are the Odds? Improving the foundations of Statistical Model Checking](https://mlsanity.com/paper/2404.05424/) (2024)
- [Learning Robust Policies for Uncertain Parametric Markov Decision Processes](https://mlsanity.com/paper/2312.06344/) (2023)
- [Multi-Objective Approaches to Markov Decision Processes with Uncertain Transition Parameters](https://mlsanity.com/paper/1710.08986/) (2017)

## Built on by (most cited first, 7 of 7)

- [Multi-Environment POMDPs: Discrete Model Uncertainty Under Partial Observability](https://mlsanity.com/paper/2510.23744/) (2025)
- [Data-Efficient Safe Policy Improvement Using Parametric Structure](https://mlsanity.com/paper/2507.15532/) (2025)
- [Robust Shielding for Safe Reinforcement Learning](https://mlsanity.com/paper/2606.00270/) (2026)
- [Efficient Solution and Learning of Robust Factored MDPs](https://mlsanity.com/paper/2508.00707/) (2025)
- [Probabilistic Performance Guarantees for Multi-Task Reinforcement Learning](https://mlsanity.com/paper/2602.02098/) (2026)
- [Robust Parameter Learning for Uncertain MDPs](https://mlsanity.com/paper/2605.01339/) (2026)
- [Robust Counterfactuals for Markov Decision Processes under Statistical and Causal Uncertainties](https://mlsanity.com/paper/2610.20210/) (2026)

## Abstract (arXiv)

We present a data-driven approach for producing policies that are provably robust across unknown stochastic environments. Existing approaches can learn models of a single environment as an interval Markov decision processes (IMDP) and produce a robust policy with a probably approximately correct (PAC) guarantee on its performance. However these are unable to reason about the impact of environmental parameters underlying the uncertainty. We propose a framework based on parametric Markov decision processes (MDPs) with unknown distributions over parameters. We learn and analyse IMDPs for a set of unknown sample environments induced by parameters. The key challenge is then to produce meaningful performance guarantees that combine the two layers of uncertainty: (1) multiple environments induced by parameters with an unknown distribution; (2) unknown induced environments which are approximated by IMDPs. We present a novel approach based on scenario optimisation that yields a single PAC guarantee quantifying the risk level for which a specified performance level can be assured in unseen environments, plus a means to trade-off risk and performance. We implement and evaluate our framework using multiple robust policy generation methods on a range of benchmarks. We show that our approach produces tight bounds on a policy's performance with high confidence.