# Bridging the Gap: Providing Post-Hoc Symbolic Explanations for Sequential Decision-Making Problems with Inscrutable Representations

arXiv: https://arxiv.org/abs/2002.01080 · HTML: https://mlsanity.com/paper/2002.01080/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Feb 2020.

## Built on by (most cited first, 30 of 32)

- [Logic-Based Explainability in Machine Learning](https://mlsanity.com/paper/2211.00541/) (2022)
- [Bridging the Human-AI Knowledge Gap: Concept Discovery and Transfer in AlphaZero](https://mlsanity.com/paper/2310.16410/) (2023)
- [Symbols as a Lingua Franca for Bridging Human-AI Chasm for Explainable and Advisable AI Systems](https://mlsanity.com/paper/2109.09904/) (2021)
- [Theory of Mind abilities of Large Language Models in Human-Robot Interaction : An Illusion?](https://mlsanity.com/paper/2401.05302/) (2024)
- [State2Explanation: Concept-Based Explanations to Benefit Agent Learning and User Understanding](https://mlsanity.com/paper/2309.12482/) (2023)
- [Natural Language Reinforcement Learning](https://mlsanity.com/paper/2402.07157/) (2024)
- [Discovering User-Interpretable Capabilities of Black-Box Planning Agents](https://mlsanity.com/paper/2107.13668/) (2021)
- [Explainable Multi-Agent Reinforcement Learning for Temporal Queries](https://mlsanity.com/paper/2305.10378/) (2023)
- [Symbol Guided Hindsight Priors for Reward Learning from Human Preferences](https://mlsanity.com/paper/2210.09151/) (2022)
- [Relative Behavioral Attributes: Filling the Gap between Symbolic Goal Specification and Reward Learning from Human Preferences](https://mlsanity.com/paper/2210.15906/) (2022)
- [Autonomous Capability Assessment of Sequential Decision-Making Systems in Stochastic Settings (Extended Version)](https://mlsanity.com/paper/2306.04806/) (2023)
- [Enabling MCTS Explainability for Sequential Planning Through Computation Tree Logic](https://mlsanity.com/paper/2407.10820/) (2024)
- [Experiential Explanations for Reinforcement Learning](https://mlsanity.com/paper/2210.04723/) (2022)
- [Natural Language Reinforcement Learning](https://mlsanity.com/paper/2411.14251/) (2024)
- [A Mental-Model Centric Landscape of Human-AI Symbiosis](https://mlsanity.com/paper/2202.09447/) (2022)
- [Towards customizable reinforcement learning agents: Enabling preference specification through online vocabulary expansion](https://mlsanity.com/paper/2210.15096/) (2022)
- [Exploiting Unlabeled Data for Feedback Efficient Human Preference based Reinforcement Learning](https://mlsanity.com/paper/2302.08738/) (2023)
- [Data Driven Reward Initialization for Preference based Reinforcement Learning](https://mlsanity.com/paper/2302.08733/) (2023)
- [ADESSE: Advice Explanations in Complex Repeated Decision-Making Environments](https://mlsanity.com/paper/2405.20705/) (2024)
- [Nonparametric Identification of Latent Concepts](https://mlsanity.com/paper/2510.00136/) (2025)
- [A State Augmentation based approach to Reinforcement Learning from Human Preferences](https://mlsanity.com/paper/2302.08734/) (2023)
- [Explainable AI Systems Must Be Contestable: Here's How to Make It Happen](https://mlsanity.com/paper/2506.01662/) (2025)
- [Goal Alignment: A Human-Aware Account of Value Alignment Problem](https://mlsanity.com/paper/2302.00813/) (2023)
- [Contrastive Explainable Clustering with Differential Privacy](https://mlsanity.com/paper/2406.04610/) (2024)
- [Explainable Reinforcement Learning Agents Using World Models](https://mlsanity.com/paper/2505.08073/) (2025)
- [On the Role of Domain Experts in Creating Effective Tutoring Systems](https://mlsanity.com/paper/2510.01432/) (2025)
- [BXRL: Behavior-Explainable Reinforcement Learning](https://mlsanity.com/paper/2603.23738/) (2026)
- [Methods and Mechanisms for Interactive Novelty Handling in Adversarial Environments](https://mlsanity.com/paper/2302.14208/) (2023)
- [Explainable Human-AI Interaction: A Planning Perspective](https://mlsanity.com/paper/2405.15804/) (2024)
- [Towards Probabilistic Planning of Explanations for Robot Navigation](https://mlsanity.com/paper/2411.05022/) (2024)

## Abstract (arXiv)

As increasingly complex AI systems are introduced into our daily lives, it becomes important for such systems to be capable of explaining the rationale for their decisions and allowing users to contest these decisions. A significant hurdle to allowing for such explanatory dialogue could be the vocabulary mismatch between the user and the AI system. This paper introduces methods for providing contrastive explanations in terms of user-specified concepts for sequential decision-making settings where the system's model of the task may be best represented as an inscrutable model. We do this by building partial symbolic models of a local approximation of the task that can be leveraged to answer the user queries. We test these methods on a popular Atari game (Montezuma's Revenge) and variants of Sokoban (a well-known planning benchmark) and report the results of user studies to evaluate whether people find explanations generated in this form useful.