# Logical Explanations for Deep Relational Machines Using Relevance Information

arXiv: https://arxiv.org/abs/1807.00595 · HTML: https://mlsanity.com/paper/1807.00595/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jul 2018.

## Built on by (most cited first, 9 of 9)

- [Incorporating Domain Knowledge into Deep Neural Networks](https://mlsanity.com/paper/2103.00180/) (2021)
- [On the Computational Intelligibility of Boolean Classifiers](https://mlsanity.com/paper/2104.06172/) (2021)
- [To trust or not to trust an explanation: using LEAF to evaluate local linear XAI methods](https://mlsanity.com/paper/2106.00461/) (2021)
- [When a Relation Tells More Than a Concept: Exploring and Evaluating Classifier Decisions with CoReX](https://mlsanity.com/paper/2405.01661/) (2024)
- [BIRDNet: Mining and Encoding Boolean Implication Knowledge Graphs as Interpretable Deep Neural Networks](https://mlsanity.com/paper/2605.28739/) (2026)
- [A Review of Some Techniques for Inclusion of Domain-Knowledge into Deep Neural Networks](https://mlsanity.com/paper/2107.10295/) (2021)
- [Incorporation of Deep Neural Network & Reinforcement Learning with Domain Knowledge](https://mlsanity.com/paper/2107.14613/) (2021)
- [Composition of Relational Features with an Application to Explaining Black-Box Predictors](https://mlsanity.com/paper/2206.00738/) (2022)
- [Saliency Methods are Encoders: Analysing Logical Relations Towards Interpretation](https://mlsanity.com/paper/2412.16204/) (2024)

## Abstract (arXiv)

Our interest in this paper is in the construction of symbolic explanations for predictions made by a deep neural network. We will focus attention on deep relational machines (DRMs, first proposed by H. Lodhi). A DRM is a deep network in which the input layer consists of Boolean-valued functions (features) that are defined in terms of relations provided as domain, or background, knowledge. Our DRMs differ from those proposed by Lodhi, which use an Inductive Logic Programming (ILP) engine to first select features (we use random selections from a space of features that satisfies some approximate constraints on logical relevance and non-redundancy). But why do the DRMs predict what they do? One way of answering this is the LIME setting, in which readable proxies for a black-box predictor. The proxies are intended only to model the predictions of the black-box in local regions of the instance-space. But readability alone may not enough: to be understandable, the local models must use relevant concepts in an meaningful manner. We investigate the use of a Bayes-like approach to identify logical proxies for local predictions of a DRM. We show: (a) DRM's with our randomised propositionalization method achieve state-of-the-art predictive performance; (b) Models in first-order logic can approximate the DRM's prediction closely in a small local region; and (c) Expert-provided relevance information can play the role of a prior to distinguish between logical explanations that perform equivalently on prediction alone.