# A Model for Intelligible Interaction Between Agents That Predict and Explain

arXiv: https://arxiv.org/abs/2301.01819 · HTML: https://mlsanity.com/paper/2301.01819/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 49 references that are in the map.
- First posted on arXiv 4 Jan 2023.

## Builds on (most cited first, 15 of 15)

- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](https://mlsanity.com/paper/2103.11251/) (2021)
- [Acquisition of Chess Knowledge in AlphaZero](https://mlsanity.com/paper/2111.09259/) (2021)
- [Right for the Right Concept: Revising Neuro-Symbolic Concepts by Interacting with their Explanations](https://mlsanity.com/paper/2011.12854/) (2020)
- [On scientific understanding with artificial intelligence](https://mlsanity.com/paper/2204.01467/) (2022)
- [Human-Centered Concept Explanations for Neural Networks](https://mlsanity.com/paper/2202.12451/) (2022)
- [A Grounded Interaction Protocol for Explainable Artificial Intelligence](https://mlsanity.com/paper/1903.02409/) (2019)
- [Beneficial and Harmful Explanatory Machine Learning](https://mlsanity.com/paper/2009.06410/) (2020)
- [Explainability Is in the Mind of the Beholder: Establishing the Foundations of Explainable Artificial Intelligence](https://mlsanity.com/paper/2112.14466/) (2021)
- [Inclusion of Domain-Knowledge into GNNs using Mode-Directed Inverse Entailment](https://mlsanity.com/paper/2105.10709/) (2021)
- [Explanatory machine learning for sequential human teaching](https://mlsanity.com/paper/2205.10250/) (2022)
- [Constructing and Evaluating an Explainable Model for COVID-19 Diagnosis from Chest X-rays](https://mlsanity.com/paper/2012.10787/) (2020)

## Built on by (most cited first, 2 of 2)

- [Multi-Turn Human-LLM Interaction Through the Lens of a Two-Way Intelligibility Protocol](https://mlsanity.com/paper/2410.20600/) (2024)
- [Structured Program Synthesis using LLMs: Results and Insights from the IPARC Challenge](https://mlsanity.com/paper/2506.13820/) (2025)

## Abstract (arXiv)

Machine Learning (ML) has emerged as a powerful form of data modelling with widespread applicability beyond its roots in the design of autonomous agents. However, relatively little attention has been paid to the interaction between people and ML systems. In this paper we view interaction between humans and ML systems within the broader context of communication between agents capable of prediction and explanation. We formalise the interaction model by taking agents to be automata with some special characteristics and define a protocol for communication between such agents. We define One- and Two-Way Intelligibility as properties that emerge at run-time by execution of the protocol. The formalisation allows us to identify conditions under which run-time sequences are bounded, and identify conditions under which the protocol can correctly implement an axiomatic specification of intelligible interaction between a human and an ML system. We also demonstrate using the formal model to: (a) identify instances of One- and Two-Way Intelligibility in literature reports on humans interacting with ML systems providing logic-based explanations, as is done in Inductive Logic Programming (ILP); and (b) map interactions between humans and machines in an elaborate natural-language based dialogue-model to One- or Two-Way Intelligible interactions in the formal model.