mlsanityData to 14 Oct 2026

Interpretable Off-Policy Evaluation in Reinforcement Learning by Highlighting Influential Transitions

arXiv 2002.03478 ↗First posted 10 Feb 2020cs.LG stat.MLcited by 41

Not on the map. Why?

Abstract (arXiv)

Off-policy evaluation in reinforcement learning offers the chance of using observational data to improve future outcomes in domains such as healthcare and education, but safe deployment in high stakes settings requires ways of assessing its validity. Traditional measures such as confidence intervals may be insufficient due to noise, limited data and confounding. In this paper we develop a method that could serve as a hybrid human-AI system, to enable human experts to analyze the validity of policy evaluation estimates. This is accomplished by highlighting observations in the data whose removal will have a large effect on the OPE estimate, and formulating a set of rules for choosing which ones to present to domain experts for validation. We develop methods to compute exactly the influence functions for fitted Q-evaluation with two different function classes: kernel-based and linear least squares, as well as importance sampling methods. Experiments on medical simulations and real-world intensive care unit data demonstrate that our method can be used to identify limitations in the evaluation process and make evaluation more robust.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). First posted on arXiv 10 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 8; 2022: 8; 2023: 7; 2024: 8; 2025: 7; 2026: 3.

41 papers.

Sort:

By area:

  1. 2023Explainable Deep Reinforcement Learning: State of the Art and Challengescited by 33
  2. 2022A Survey of Explainable Reinforcement Learningcited by 31
  3. 2021Continuous Doubly Constrained Batch Reinforcement Learningcited by 27
  4. 2021Finite-Sample Analysis of Off-Policy Natural Actor-Critic Algorithmcited by 26
  5. 2021Finite-Sample Analysis of Off-Policy Natural Actor-Critic with Linear Function Approximationcited by 22
  6. 2022The Health Gym: Synthetic Health-Related Datasets for the Development of Reinforcement Learning Algorithmscited by 19
  7. 2021Finite-Sample Analysis of Off-Policy TD-Learning via Generalized Bellman Operatorscited by 17
  8. 2022Sample Complexity of Nonparametric Off-Policy Evaluation on Low-Dimensional Manifolds using Deep Networkscited by 12
  9. 2022Generating Synthetic Clinical Data that Capture Class Imbalanced Distributions with Generative Adversarial Networks: Example using Antiretroviral Therapy for HIVcited by 12
  10. 2021The Medkit-Learn(ing) Environment: Medical Decision Modelling through Simulationcited by 11
  11. 2023Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examplescited by 9
  12. 2022Reinforcement Learning in Practice: Opportunities and Challengescited by 6
29 more
  1. 2025Translate Policy to Language: Flow Matching Generated Rewards for LLM Explanationscited by 6
  2. 2021Synthetic Acute Hypotension and Sepsis Datasets Based on MIMIC-III and Published as Part of the Health Gym Projectcited by 5
  3. 2023HOPE: Human-Centric Off-Policy Evaluation for E-Learning and Healthcarecited by 5
  4. 2024Reinforcement Learning in Dynamic Treatment Regimes Needs Critical Reexaminationcited by 5
  5. 2022Generalizing Off-Policy Evaluation From a Causal Perspective For Sequential Decision-Makingcited by 4
  6. 2024Data Poisoning Attacks on Off-Policy Policy Evaluation Methodscited by 4
  7. 2025PERRY: Policy Evaluation with Confidence Intervals using Auxiliary Datacited by 4
  8. 2021State Relevance for Off-Policy Evaluationcited by 3
  9. 2023The Unintended Consequences of Discount Regularization: Improving Regularization in Certainty Equivalence Reinforcement Learningcited by 3
  10. 2022Mitigating Health Data Poverty: Generative Approaches versus Resampling for Time-series Clinical Datacited by 2
  11. 2022Relative Sparsity for Medical Decision Problemscited by 2
  12. 2024How Consistent are Clinicians? Evaluating the Predictability of Sepsis Disease Progression with Dynamics Modelscited by 2
  13. 2024Empowering Clinicians with Medical Decision Transformers: A Framework for Sepsis Treatmentcited by 2
  14. 2025Interactive Explanations for Reinforcement-Learning Agentscited by 2
  15. 2026SymbXRL: Symbolic Explainable Deep Reinforcement Learning for Mobile Networkscited by 2
  16. 2021Case-based off-policy policy evaluation using prototype learningcited by 1
  17. 2024Which Experiences Are Influential for RL Agents? Efficiently Estimating The Influence of Experiencescited by 1
  18. 2024Identifying Differential Patient Care Through Inverse Intent Inferencecited by 1
  19. 2024Concept-driven Off Policy Evaluationcited by 1
  20. 2025Limits of Generative Pre-Training in Structured EMR Trajectories with Irregular Samplingcited by 1
  21. 2023ASQ-IT: Interactive Explanations for Reinforcement-Learning Agentscited by 0
  22. 2023Inference for relative sparsitycited by 0
  23. 2023Deep Attention Q-Network for Personalized Treatment Recommendationcited by 0
  24. 2024Closed-loop Teaching via Demonstrations to Improve Policy Transparencycited by 0
  25. 2025From What Ifs to Insights: Counterfactuals in Causal Inference vs. Explainable AIcited by 0
  26. 2025Treatment, evidence, imitation, and chatcited by 0
  27. 2025Which Rewards Matter? Reward Selection for Reinforcement Learning under Limited Feedbackcited by 0
  28. 2026Evaluation metrics for temporal preservation in synthetic longitudinal patient datacited by 0
  29. 2026Data Attribution in Adaptive Learningcited by 0