# Test Where Decisions Matter: Importance-driven Testing for Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2411.07700 · HTML: https://mlsanity.com/paper/2411.07700/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 17 of its 33 references that are in the map.
- First posted on arXiv 12 Nov 2024.

## Placed in

- Deep reinforcement learning methods / [Safe and constrained RL](https://mlsanity.com/deeprl/safe-and-constrained-rl/)

## Builds on (most cited first, 17 of 17)

- [Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2002.00444/) (2020)
- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [Doubly Robust Off-policy Value Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1511.03722/) (2015)
- [Explainability in Deep Reinforcement Learning](https://mlsanity.com/paper/2008.06693/) (2020)
- [A Review of Off-Policy Evaluation in Reinforcement Learning](https://mlsanity.com/paper/2212.06355/) (2022)
- [Deep Clustering: A Comprehensive Survey](https://mlsanity.com/paper/2210.04142/) (2022)
- [Universal Off-Policy Evaluation](https://mlsanity.com/paper/2104.12820/) (2021)
- [Safe Reinforcement Learning via Probabilistic Logic Shields](https://mlsanity.com/paper/2303.03226/) (2023)
- [Importance-Driven Deep Learning System Testing](https://mlsanity.com/paper/2002.03433/) (2020)
- [Robust Control for Dynamical Systems With Non-Gaussian Noise via Formal Abstractions](https://mlsanity.com/paper/2301.01526/) (2023)
- [TEMPEST -- Synthesis Tool for Reactive Systems and Shields in Probabilistic Environments](https://mlsanity.com/paper/2105.12588/) (2021)
- [Search-Based Testing of Reinforcement Learning](https://mlsanity.com/paper/2205.04887/) (2022)
- [A Search-Based Testing Approach for Deep Reinforcement Learning Agents](https://mlsanity.com/paper/2206.07813/) (2022)
- [Testing of Deep Reinforcement Learning Agents with Surrogate Models](https://mlsanity.com/paper/2305.12751/) (2023)
- [MDPFuzz: Testing Models Solving Markov Decision Processes](https://mlsanity.com/paper/2112.02807/) (2021)
- [Ranking Policy Decisions](https://mlsanity.com/paper/2008.13607/) (2020)
- [Causal policy ranking](https://mlsanity.com/paper/2111.08415/) (2021)

## Built on by (most cited first, 2 of 2)

- [Explainably Safe Reinforcement Learning](https://mlsanity.com/paper/2606.04634/) (2026)
- [Property-driven Causal Abstractions for Markov Decision Processes](https://mlsanity.com/paper/2607.26787/) (2026)

## Abstract (arXiv)

In many Deep Reinforcement Learning (RL) problems, decisions in a trained policy vary in significance for the expected safety and performance of the policy. Since RL policies are very complex, testing efforts should concentrate on states in which the agent's decisions have the highest impact on the expected outcome. In this paper, we propose a novel model-based method to rigorously compute a ranking of state importance across the entire state space. We then focus our testing efforts on the highest-ranked states. In this paper, we focus on testing for safety. However, the proposed methods can be easily adapted to test for performance. In each iteration, our testing framework computes optimistic and pessimistic safety estimates. These estimates provide lower and upper bounds on the expected outcomes of the policy execution across all modeled states in the state space. Our approach divides the state space into safe and unsafe regions upon convergence, providing clear insights into the policy's weaknesses. Two important properties characterize our approach. (1) Optimal Test-Case Selection: At any time in the testing process, our approach evaluates the policy in the states that are most critical for safety. (2) Guaranteed Safety: Our approach can provide formal verification guarantees over the entire state space by sampling only a fraction of the policy. Any safety properties assured by the pessimistic estimate are formally proven to hold for the policy. We provide a detailed evaluation of our framework on several examples, showing that our method discovers unsafe policy behavior with low testing effort.