mlsanityData to 14 Oct 2026

Establishing Appropriate Trust via Critical States

arXiv 1810.08174 ↗First posted 18 Oct 2018cs.ROcited by 48

Not on the map. Why?

Abstract (arXiv)

In order to effectively interact with or supervise a robot, humans need to have an accurate mental model of its capabilities and how it acts. Learned neural network policies make that particularly challenging. We propose an approach for helping end-users build a mental model of such policies. Our key observation is that for most tasks, the essence of the policy is captured in a few critical states: states in which it is very important to take a certain action. Our user studies show that if the robot shows a human what its understanding of the task's critical states is, then the human can make a more informed decision about whether to deploy the policy, and if she does deploy it, when she needs to take control from it at execution time.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). First posted on arXiv 18 Oct 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 7; 2022: 9; 2023: 12; 2024: 8; 2025: 10; 2026: 2.

48 papers.

Sort:

By area:

  1. 2022Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanationscited by 85
  2. 2021Counterfactual State Explanations for Reinforcement Learning Agents via Generative Deep Learningcited by 39
  3. 2023Explainable Deep Reinforcement Learning: State of the Art and Challengescited by 33
  4. 2022A Survey of Explainable Reinforcement Learningcited by 31
  5. 2021Explainable Reinforcement Learning for Broad-XAI: A Conceptual Framework and Surveycited by 24
  6. 2023A Systematic Review on Fostering Appropriate Trust in Human-AI Interactioncited by 21
  7. 2023GANterfactual-RL: Understanding Reinforcement Learning Agents' Strategies through Visual Counterfactual Explanationscited by 17
  8. 2023Explaining Reinforcement Learning Agents Through Counterfactual Action Outcomescited by 14
  9. 2024Understanding Individual Agent Importance in Multi-Agent System via Counterfactual Reasoningcited by 11
  10. 2022Explaining Reinforcement Learning Policies through Counterfactual Trajectoriescited by 7
  11. 2025Democratizing Diplomacy: A Harness for Evaluating Any Large Language Model on Full-Press Diplomacycited by 7
  12. 2022Reasoning about Counterfactuals to Improve Human Inverse Reinforcement Learningcited by 6
36 more
  1. 2024Conformalized Teleoperation: Confidently Mapping Human Inputs to High-Dimensional Robot Actionscited by 6
  2. 2025Translate Policy to Language: Flow Matching Generated Rewards for LLM Explanationscited by 6
  3. 2025A Statistical Case Against Empirical Human-AI Alignmentcited by 6
  4. 2022Local Explanations for Reinforcement Learningcited by 5
  5. 2023Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challengescited by 5
  6. 2021Show Me What You Can Do: Capability Calibration on Reachable Workspace for Human-Robot Collaborationcited by 4
  7. 2023Adaptive Failure Search Using Critical States from Domain Expertscited by 4
  8. 2024Crafting Desirable Climate Trajectories with RL Explored Socio-Environmental Simulationscited by 4
  9. 2021Explaining Reward Functions to Humans for Better Human-Robot Collaborationcited by 3
  10. 2025TalkToAgent: A Human-centric Explanation of Reinforcement Learning Agents with Large Language Modelscited by 3
  11. 2021AutoPreview: A Framework for Autopilot Behavior Understandingcited by 2
  12. 2021Identifying Reasoning Flaws in Planning-Based RL Using Tree Explanationscited by 2
  13. 2022Reward Learning with Trees: Methods and Evaluationcited by 2
  14. 2023Safety Margins for Reinforcement Learningcited by 2
  15. 2023Multi-Agent Strategy Explanations for Human-Robot Collaborationcited by 2
  16. 2024REACT: Revealing Evolutionary Action Consequence Trajectories for Interpretable Reinforcement Learningcited by 2
  17. 2024Navigating Trade-offs: Policy Summarization for Multi-Objective Reinforcement Learningcited by 2
  18. 2025Co-Activation Graph Analysis of Safety-Verified and Explainable Deep Reinforcement Learning Policiescited by 2
  19. 2025Interactive Explanations for Reinforcement-Learning Agentscited by 2
  20. 2022Causal Robot Communication Inspired by Observational Learning Insightscited by 1
  21. 2022(When) Are Contrastive Explanations of Reinforcement Learning Helpful?cited by 1
  22. 2024Explaining Learned Reward Functions with Counterfactual Trajectoriescited by 1
  23. 2025Surrogate Fitness Metrics for Interpretable Reinforcement Learningcited by 1
  24. 2025MACIE: Multi-Agent Causal Intelligence Explainer for Collective Behavior Understandingcited by 1
  25. 2021Building Mental Models through Preview of Autopilot Behaviorscited by 0
  26. 2022Global and Local Analysis of Interestingness for Competency-Aware Deep Reinforcement Learningcited by 0
  27. 2023ASQ-IT: Interactive Explanations for Reinforcement-Learning Agentscited by 0
  28. 2023Conveying Autonomous Robot Capabilities through Contrasting Behaviour Summariescited by 0
  29. 2023Learning Interpretable Models of Aircraft Handling Behaviour by Reinforcement Learning from Human Feedbackcited by 0
  30. 2023IxDRL: A Novel Explainable Deep Reinforcement Learning Toolkit based on Analyses of Interestingnesscited by 0
  31. 2024Closed-loop Teaching via Demonstrations to Improve Policy Transparencycited by 0
  32. 2024Criticality and Safety Margins for Reinforcement Learningcited by 0
  33. 2025"What are my options?": Explaining RL Agents with Diverse Near-Optimal Alternatives (Extended)cited by 0
  34. 2025MAGIC-MASK: Multi-Agent Guided Inter-Agent Collaboration with Mask-Based Explainability for Reinforcement Learningcited by 0
  35. 2026Distilling Deep Reinforcement Learning into Interpretable Fuzzy Rules: An Explainable AI Frameworkcited by 0
  36. 2026Where Small Mistakes Matter: Quantifying State Criticality for Imitation Learningcited by 0