# Interpretable Off-Policy Evaluation in Reinforcement Learning by Highlighting Influential Transitions

arXiv: https://arxiv.org/abs/2002.03478 · HTML: https://mlsanity.com/paper/2002.03478/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Feb 2020.

## Built on by (most cited first, 30 of 41)

- [Explainable Deep Reinforcement Learning: State of the Art and Challenges](https://mlsanity.com/paper/2301.09937/) (2023)
- [A Survey of Explainable Reinforcement Learning](https://mlsanity.com/paper/2202.08434/) (2022)
- [Continuous Doubly Constrained Batch Reinforcement Learning](https://mlsanity.com/paper/2102.09225/) (2021)
- [Finite-Sample Analysis of Off-Policy Natural Actor-Critic Algorithm](https://mlsanity.com/paper/2102.09318/) (2021)
- [Finite-Sample Analysis of Off-Policy Natural Actor-Critic with Linear Function Approximation](https://mlsanity.com/paper/2105.12540/) (2021)
- [The Health Gym: Synthetic Health-Related Datasets for the Development of Reinforcement Learning Algorithms](https://mlsanity.com/paper/2203.06369/) (2022)
- [Finite-Sample Analysis of Off-Policy TD-Learning via Generalized Bellman Operators](https://mlsanity.com/paper/2106.12729/) (2021)
- [Sample Complexity of Nonparametric Off-Policy Evaluation on Low-Dimensional Manifolds using Deep Networks](https://mlsanity.com/paper/2206.02887/) (2022)
- [Generating Synthetic Clinical Data that Capture Class Imbalanced Distributions with Generative Adversarial Networks: Example using Antiretroviral Therapy for HIV](https://mlsanity.com/paper/2208.08655/) (2022)
- [The Medkit-Learn(ing) Environment: Medical Decision Modelling through Simulation](https://mlsanity.com/paper/2106.04240/) (2021)
- [Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examples](https://mlsanity.com/paper/2310.07747/) (2023)
- [Reinforcement Learning in Practice: Opportunities and Challenges](https://mlsanity.com/paper/2202.11296/) (2022)
- [Translate Policy to Language: Flow Matching Generated Rewards for LLM Explanations](https://mlsanity.com/paper/2502.12530/) (2025)
- [Synthetic Acute Hypotension and Sepsis Datasets Based on MIMIC-III and Published as Part of the Health Gym Project](https://mlsanity.com/paper/2112.03914/) (2021)
- [HOPE: Human-Centric Off-Policy Evaluation for E-Learning and Healthcare](https://mlsanity.com/paper/2302.09212/) (2023)
- [Reinforcement Learning in Dynamic Treatment Regimes Needs Critical Reexamination](https://mlsanity.com/paper/2405.18556/) (2024)
- [Generalizing Off-Policy Evaluation From a Causal Perspective For Sequential Decision-Making](https://mlsanity.com/paper/2201.08262/) (2022)
- [Data Poisoning Attacks on Off-Policy Policy Evaluation Methods](https://mlsanity.com/paper/2404.04714/) (2024)
- [PERRY: Policy Evaluation with Confidence Intervals using Auxiliary Data](https://mlsanity.com/paper/2507.20068/) (2025)
- [State Relevance for Off-Policy Evaluation](https://mlsanity.com/paper/2109.06310/) (2021)
- [The Unintended Consequences of Discount Regularization: Improving Regularization in Certainty Equivalence Reinforcement Learning](https://mlsanity.com/paper/2306.11208/) (2023)
- [Mitigating Health Data Poverty: Generative Approaches versus Resampling for Time-series Clinical Data](https://mlsanity.com/paper/2210.13958/) (2022)
- [Relative Sparsity for Medical Decision Problems](https://mlsanity.com/paper/2211.16566/) (2022)
- [How Consistent are Clinicians? Evaluating the Predictability of Sepsis Disease Progression with Dynamics Models](https://mlsanity.com/paper/2404.07148/) (2024)
- [Empowering Clinicians with Medical Decision Transformers: A Framework for Sepsis Treatment](https://mlsanity.com/paper/2407.19380/) (2024)
- [Interactive Explanations for Reinforcement-Learning Agents](https://mlsanity.com/paper/2504.05393/) (2025)
- [SymbXRL: Symbolic Explainable Deep Reinforcement Learning for Mobile Networks](https://mlsanity.com/paper/2601.22024/) (2026)
- [Case-based off-policy policy evaluation using prototype learning](https://mlsanity.com/paper/2111.11113/) (2021)
- [Which Experiences Are Influential for RL Agents? Efficiently Estimating The Influence of Experiences](https://mlsanity.com/paper/2405.14629/) (2024)
- [Identifying Differential Patient Care Through Inverse Intent Inference](https://mlsanity.com/paper/2411.07372/) (2024)

## Abstract (arXiv)

Off-policy evaluation in reinforcement learning offers the chance of using observational data to improve future outcomes in domains such as healthcare and education, but safe deployment in high stakes settings requires ways of assessing its validity. Traditional measures such as confidence intervals may be insufficient due to noise, limited data and confounding. In this paper we develop a method that could serve as a hybrid human-AI system, to enable human experts to analyze the validity of policy evaluation estimates. This is accomplished by highlighting observations in the data whose removal will have a large effect on the OPE estimate, and formulating a set of rules for choosing which ones to present to domain experts for validation. We develop methods to compute exactly the influence functions for fitted Q-evaluation with two different function classes: kernel-based and linear least squares, as well as importance sampling methods. Experiments on medical simulations and real-world intensive care unit data demonstrate that our method can be used to identify limitations in the evaluation process and make evaluation more robust.