# Scalable Safety-Critical Policy Evaluation with Accelerated Rare Event Sampling

arXiv: https://arxiv.org/abs/2106.10566 · HTML: https://mlsanity.com/paper/2106.10566/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 48 references that are in the map.
- First posted on arXiv 19 Jun 2021.

## Builds on (most cited first, 23 of 23)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Masked Autoregressive Flow for Density Estimation](https://mlsanity.com/paper/1705.07057/) (2017)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design](https://mlsanity.com/paper/2012.02096/) (2020)
- [Safe and Efficient Off-Policy Reinforcement Learning](https://mlsanity.com/paper/1606.02647/) (2016)
- [Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations](https://mlsanity.com/paper/2003.08938/) (2020)
- [Action Robust Reinforcement Learning and Applications in Continuous Control](https://mlsanity.com/paper/1901.09184/) (2019)
- [Autonomous Agents Modelling Other Agents: A Comprehensive Survey and Open Problems](https://mlsanity.com/paper/1709.08071/) (2017)
- [Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling](https://mlsanity.com/paper/1906.03393/) (2019)
- [Scalable End-to-End Autonomous Vehicle Testing via Rare-event Simulation](https://mlsanity.com/paper/1811.00145/) (2018)
- [A Survey of Algorithms for Black-Box Safety Validation of Cyber-Physical Systems](https://mlsanity.com/paper/2005.02979/) (2020)
- [Multimodal Safety-Critical Scenarios Generation for Decision-Making Algorithms Evaluation](https://mlsanity.com/paper/2009.08311/) (2020)
- [Accelerated Evaluation of Automated Vehicles Safety in Lane Change Scenarios Based on Importance Sampling Techniques](https://mlsanity.com/paper/1605.04965/) (2016)
- [Context-Aware Safe Reinforcement Learning for Non-Stationary Environments](https://mlsanity.com/paper/2101.00531/) (2021)
- [The Variational Gaussian Process](https://mlsanity.com/paper/1511.06499/) (2015)
- [Importance Resampling for Off-policy Prediction](https://mlsanity.com/paper/1906.04328/) (2019)
- [Detecting Overfitting via Adversarial Examples](https://mlsanity.com/paper/1903.02380/) (2019)
- [Scalable Autonomous Vehicle Safety Validation through Dynamic Programming and Scene Decomposition](https://mlsanity.com/paper/2004.06801/) (2020)
- [Deep Probabilistic Accelerated Evaluation: A Robust Certifiable Rare-Event Simulation Methodology for Black-Box Safety-Critical Systems](https://mlsanity.com/paper/2006.15722/) (2020)
- [Off-Policy Evaluation of Probabilistic Identity Data in Lookalike Modeling](https://mlsanity.com/paper/1901.05560/) (2019)

## Built on by (most cited first, 2 of 2)

- [CaDRE: Controllable and Diverse Generation of Safety-Critical Driving Scenarios using Real-World Trajectories](https://mlsanity.com/paper/2403.13208/) (2024)
- [Continual Vision-based Reinforcement Learning with Group Symmetries](https://mlsanity.com/paper/2210.12301/) (2022)

## Abstract (arXiv)

Evaluating rare but high-stakes events is one of the main challenges in obtaining reliable reinforcement learning policies, especially in large or infinite state/action spaces where limited scalability dictates a prohibitively large number of testing iterations. On the other hand, a biased or inaccurate policy evaluation in a safety-critical system could potentially cause unexpected catastrophic failures during deployment. This paper proposes the Accelerated Policy Evaluation (APE) method, which simultaneously uncovers rare events and estimates the rare event probability in Markov decision processes. The APE method treats the environment nature as an adversarial agent and learns towards, through adaptive importance sampling, the zero-variance sampling distribution for the policy evaluation. Moreover, APE is scalable to large discrete or continuous spaces by incorporating function approximators. We investigate the convergence property of APE in the tabular setting. Our empirical studies show that APE can estimate the rare event probability with a smaller bias while only using orders of magnitude fewer samples than baselines in multi-agent and single-agent environments.