mlsanityData to 7 Oct 2026

Explaining Reinforcement Learning Policies through Counterfactual Trajectories

arXiv 2201.12462 ↗First posted 29 Jan 2022cs.LG cs.AI cs.HCcited by 7

Not on the map. Why?

Abstract (arXiv)

In order for humans to confidently decide where to employ RL agents for real-world tasks, a human developer must validate that the agent will perform well at test-time. Some policy interpretability methods facilitate this by capturing the policy's decision making in a set of agent rollouts. However, even the most informative trajectories of training time behavior may give little insight into the agent's behavior out of distribution. In contrast, our method conveys how the agent performs under distribution shifts by showing the agent's behavior across a wider trajectory distribution. We generate these trajectories by guiding the agent to more diverse unseen states and showing the agent's behavior there. In a user study, we demonstrate that our method enables users to score better than baseline methods on one of two agent validation tasks.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 20 of its 25 references that are in the map. First posted on arXiv 29 Jan 2022.

Builds on

20 of 25 references.

References · 20

  1. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  2. 2018DeepMind Control Suitecited by 935
  3. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  4. 2019Solving Rubik's Cube with a Robot Handcited by 699
  5. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  6. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  7. 2018Metrics for Explainable AI: Challenges and Prospectscited by 257
  8. 2015DeepDriving: Learning Affordance for Direct Perception in Autonomous Drivingcited by 229
  9. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  10. 2019Explainable Reinforcement Learning Through a Causal Lenscited by 177
  11. 2017Visualizing and Understanding Atari Agentscited by 141
  12. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
8 more
  1. 2020Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifierscited by 93
  2. 2016Sim-to-Real Robot Learning from Pixels with Progressive Netscited by 89
  3. 2019Explain Your Move: Understanding Agent Actions Using Specific and Relevant Feature Attributioncited by 50
  4. 2018Establishing Appropriate Trust via Critical Statescited by 47
  5. 2019Explaining Reinforcement Learning to Mere Mortals: An Empirical Studycited by 21
  6. 2018Measuring and Characterizing Generalization in Deep Reinforcement Learningcited by 20
  7. 2019Finding and Visualizing Weaknesses of Deep Reinforcement Learning Agentscited by 15
  8. 2019Counterfactual States for Atari Agents via Generative Deep Learningcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 1; 2024: 1; 2025: 3; 2026: 1.

7 papers.

  1. 2022Experiential Explanations for Reinforcement Learningcited by 5
  2. 2024Explaining Learned Reward Functions with Counterfactual Trajectoriescited by 1
  3. 2025Counterfactual Explanations for Continuous Action Reinforcement Learningcited by 1
  4. 2023Personalized Path Recourse for Reinforcement Learning Agentscited by 0
  5. 2025From Actions to Words: Towards Abstractive-Textual Policy Summarization in RLcited by 0
  6. 2025Know your Trajectory -- Trustworthy Reinforcement Learning deployment through Importance-Based Trajectory Analysiscited by 0
  7. 2026Evaluating RL Explainability Methods by How Much They Help Fix Bugs in Agentscited by 0