# Natural Language Interaction with Explainable AI Models

arXiv: https://arxiv.org/abs/1903.05720 · HTML: https://mlsanity.com/paper/1903.05720/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Mar 2019.

## Built on by (most cited first, 8 of 8)

- [CX-ToM: Counterfactual Explanations with Theory-of-Mind for Enhancing Human Trust in Image Recognition Models](https://mlsanity.com/paper/2109.01401/) (2021)
- [Mediators: Conversational Agents Explaining NLP Model Behavior](https://mlsanity.com/paper/2206.06029/) (2022)
- [Attention cannot be an Explanation](https://mlsanity.com/paper/2201.11194/) (2022)
- [Question Generation for Evaluating Cross-Dataset Shifts in Multi-modal Grounding](https://mlsanity.com/paper/2201.09639/) (2022)
- [Stereotype Detection in LLMs: A Multiclass, Explainable, and Benchmark-Driven Approach](https://mlsanity.com/paper/2404.01768/) (2024)
- [Discourse Analysis for Evaluating Coherence in Video Paragraph Captions](https://mlsanity.com/paper/2201.06207/) (2022)
- [OAK4XAI: Model towards Out-Of-Box eXplainable Artificial Intelligence for Digital Agriculture](https://mlsanity.com/paper/2209.15104/) (2022)
- [On the Importance of Karaka Framework in Multi-modal Grounding](https://mlsanity.com/paper/2204.04347/) (2022)

## Abstract (arXiv)

This paper presents an explainable AI (XAI) system that provides explanations for its predictions. The system consists of two key components -- namely, the prediction And-Or graph (AOG) model for recognizing and localizing concepts of interest in input data, and the XAI model for providing explanations to the user about the AOG's predictions. In this work, we focus on the XAI model specified to interact with the user in natural language, whereas the AOG's predictions are considered given and represented by the corresponding parse graphs (pg's) of the AOG. Our XAI model takes pg's as input and provides answers to the user's questions using the following types of reasoning: direct evidence (e.g., detection scores), part-based inference (e.g., detected parts provide evidence for the concept asked), and other evidences from spatio-temporal context (e.g., constraints from the spatio-temporal surround). We identify several correlations between user's questions and the XAI answers using Youtube Action dataset.