# ERIC: Extracting Relations Inferred from Convolutions

arXiv: https://arxiv.org/abs/2010.09452 · HTML: https://mlsanity.com/paper/2010.09452/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Oct 2020.

## Built on by (most cited first, 5 of 5)

- [Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks](https://mlsanity.com/paper/2207.13243/) (2022)
- [Using Logic Programming and Kernel-Grouping for Improving Interpretability of Convolutional Neural Networks](https://mlsanity.com/paper/2310.13073/) (2023)
- [Symbolic Rule Extraction from Attention-Guided Sparse Representations in Vision Transformers](https://mlsanity.com/paper/2505.06745/) (2025)
- [NeSyFOLD: Neurosymbolic Framework for Interpretable Image Classification](https://mlsanity.com/paper/2301.12667/) (2023)
- [SpIn-ViT: Designing a Sparsity-Induced Vision Transformer That Is Mechanistically Interpretable](https://mlsanity.com/paper/2608.14922/) (2026)

## Abstract (arXiv)

Our main contribution is to show that the behaviour of kernels across multiple layers of a convolutional neural network can be approximated using a logic program. The extracted logic programs yield accuracies that correlate with those of the original model, though with some information loss in particular as approximations of multiple layers are chained together or as lower layers are quantised. We also show that an extracted program can be used as a framework for further understanding the behaviour of CNNs. Specifically, it can be used to identify key kernels worthy of deeper inspection and also identify relationships with other kernels in the form of the logical rules. Finally, we make a preliminary, qualitative assessment of rules we extract from the last convolutional layer and show that kernels identified are symbolic in that they react strongly to sets of similar images that effectively divide output classes into sub-classes with distinct characteristics.