# Expressive Explanations of DNNs by Combining Concept Analysis with ILP

arXiv: https://arxiv.org/abs/2105.07371 · HTML: https://mlsanity.com/paper/2105.07371/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 35 references that are in the map.
- First posted on arXiv 16 May 2021.

## Builds on (most cited first, 13 of 13)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [Unmasking Clever Hans Predictors and Assessing What Machines Really Learn](https://mlsanity.com/paper/1902.10178/) (2019)
- [One weird trick for parallelizing convolutional neural networks](https://mlsanity.com/paper/1404.5997/) (2014)
- [On Completeness-aware Concept-Based Explanations in Deep Neural Networks](https://mlsanity.com/paper/1910.07969/) (2019)
- [Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models](https://mlsanity.com/paper/1708.08296/) (2017)
- [One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques](https://mlsanity.com/paper/1909.03012/) (2019)
- [Net2Vec: Quantifying and Explaining how Concepts are Encoded by Filters in Deep Neural Networks](https://mlsanity.com/paper/1801.03454/) (2018)
- [Logic Tensor Networks for Semantic Image Interpretation](https://mlsanity.com/paper/1705.08968/) (2017)
- [Enriching Visual with Verbal Explanations for Relational Concepts -- Combining LIME with Aleph](https://mlsanity.com/paper/1910.01837/) (2019)

## Built on by (most cited first, 11 of 11)

- [Concept Embedding Analysis: A Review](https://mlsanity.com/paper/2203.13909/) (2022)
- [Evaluating the Stability of Semantic Concept Representations in CNNs for Robust Explainability](https://mlsanity.com/paper/2304.14864/) (2023)
- [Enabling Verification of Deep Neural Networks in Perception Tasks Using Fuzzy Logic and Concept Embeddings](https://mlsanity.com/paper/2201.00572/) (2022)
- [Local Concept Embeddings for Analysis of Concept Distributions in Vision DNN Feature Spaces](https://mlsanity.com/paper/2311.14435/) (2023)
- [Revealing Similar Semantics Inside CNNs: An Interpretable Concept-based Comparison of Feature Spaces](https://mlsanity.com/paper/2305.07663/) (2023)
- [Mapping Knowledge Representations to Concepts: A Review and New Perspectives](https://mlsanity.com/paper/2301.00189/) (2022)
- [When a Relation Tells More Than a Concept: Exploring and Evaluating Classifier Decisions with CoReX](https://mlsanity.com/paper/2405.01661/) (2024)
- [Unveiling Ontological Commitment in Multi-Modal Foundation Models](https://mlsanity.com/paper/2409.17109/) (2024)
- [On Background Bias of Post-Hoc Concept Embeddings in Computer Vision DNNs](https://mlsanity.com/paper/2504.08602/) (2025)
- [CorrLoss: Integrating Co-Occurrence Domain Knowledge for Affect Recognition](https://mlsanity.com/paper/2210.17233/) (2022)
- [HyDI: A hybrid Deep Learning-Inductive Logic Programming ensemble for multi-label classification](https://mlsanity.com/paper/2609.37740/) (2026)

## Abstract (arXiv)

Explainable AI has emerged to be a key component for black-box machine learning approaches in domains with a high demand for reliability or transparency. Examples are medical assistant systems, and applications concerned with the General Data Protection Regulation of the European Union, which features transparency as a cornerstone. Such demands require the ability to audit the rationale behind a classifier's decision. While visualizations are the de facto standard of explanations, they come short in terms of expressiveness in many ways: They cannot distinguish between different attribute manifestations of visual features (e.g. eye open vs. closed), and they cannot accurately describe the influence of absence of, and relations between features. An alternative would be more expressive symbolic surrogate models. However, these require symbolic inputs, which are not readily available in most computer vision tasks. In this paper we investigate how to overcome this: We use inherent features learned by the network to build a global, expressive, verbal explanation of the rationale of a feed-forward convolutional deep neural network (DNN). The semantics of the features are mined by a concept analysis approach trained on a set of human understandable visual concepts. The explanation is found by an Inductive Logic Programming (ILP) method and presented as first-order rules. We show that our explanation is faithful to the original black-box model. The code for our experiments is available at https://github.com/mc-lovin-mlem/concept-embeddings-and-ilp/tree/ki2020.