# Using Logic Programming and Kernel-Grouping for Improving Interpretability of Convolutional Neural Networks

arXiv: https://arxiv.org/abs/2310.13073 · HTML: https://mlsanity.com/paper/2310.13073/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 38 references that are in the map.
- First posted on arXiv 19 Oct 2023.

## Builds on (most cited first, 21 of 21)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [This Looks Like That: Deep Learning for Interpretable Image Recognition](https://mlsanity.com/paper/1806.10574/) (2018)
- [Learning Explanatory Rules from Noisy Data](https://mlsanity.com/paper/1711.04574/) (2017)
- [NeurASP: Embracing Neural Networks into Answer Set Programming](https://mlsanity.com/paper/2307.07700/) (2023)
- [Extraction of Salient Sentences from Labelled Documents](https://mlsanity.com/paper/1412.6815/) (2014)
- [Interpreting CNN Knowledge via an Explanatory Graph](https://mlsanity.com/paper/1708.01785/) (2017)
- [Constraint Answer Set Programming without Grounding](https://mlsanity.com/paper/1804.11162/) (2018)
- [Neuro-Symbolic Inductive Logic Programming with Logical Neural Networks](https://mlsanity.com/paper/2112.03324/) (2021)
- [The ILASP system for Inductive Learning of Answer Set Programs](https://mlsanity.com/paper/2005.00904/) (2020)
- [Differentiable Inductive Logic Programming for Structured Examples](https://mlsanity.com/paper/2103.01719/) (2021)
- [FOLD-SE: An Efficient Rule-based Machine Learning Algorithm with Scalable Explainability](https://mlsanity.com/paper/2208.07912/) (2022)
- [Embedding Deep Networks into Visual Explanations](https://mlsanity.com/paper/1709.05360/) (2017)
- [Explaining Deep Convolutional Neural Networks via Latent Visual-Semantic Filter Attention](https://mlsanity.com/paper/2204.04601/) (2022)
- [Growing Interpretable Part Graphs on ConvNets via Multi-Shot Learning](https://mlsanity.com/paper/1611.04246/) (2016)
- [ERIC: Extracting Relations Inferred from Convolutions](https://mlsanity.com/paper/2010.09452/) (2020)
- [Relating Input Concepts to Convolutional Neural Network Decisions](https://mlsanity.com/paper/1711.08006/) (2017)
- [NeSyFOLD: Neurosymbolic Framework for Interpretable Image Classification](https://mlsanity.com/paper/2301.12667/) (2023)

## Built on by (most cited first, 4 of 4)

- [Symbolic Rule Extraction from Attention-Guided Sparse Representations in Vision Transformers](https://mlsanity.com/paper/2505.06745/) (2025)
- [A Neurosymbolic Framework for Bias Correction in Convolutional Neural Networks](https://mlsanity.com/paper/2405.15886/) (2024)
- [CFGs: Causality Constrained Counterfactual Explanations using goal-directed ASP](https://mlsanity.com/paper/2405.15956/) (2024)
- [Identifying Good Rules for Efficient SAT Encodings of Single-Constant Multiplication Using Machine Learning](https://mlsanity.com/paper/2607.21188/) (2026)

## Abstract (arXiv)

Within the realm of deep learning, the interpretability of Convolutional Neural Networks (CNNs), particularly in the context of image classification tasks, remains a formidable challenge. To this end we present a neurosymbolic framework, NeSyFOLD-G that generates a symbolic rule-set using the last layer kernels of the CNN to make its underlying knowledge interpretable. What makes NeSyFOLD-G different from other similar frameworks is that we first find groups of similar kernels in the CNN (kernel-grouping) using the cosine-similarity between the feature maps generated by various kernels. Once such kernel groups are found, we binarize each kernel group's output in the CNN and use it to generate a binarization table which serves as input data to FOLD-SE-M which is a Rule Based Machine Learning (RBML) algorithm. FOLD-SE-M then generates a rule-set that can be used to make predictions. We present a novel kernel grouping algorithm and show that grouping similar kernels leads to a significant reduction in the size of the rule-set generated by FOLD-SE-M, consequently, improving the interpretability. This rule-set symbolically encapsulates the connectionist knowledge of the trained CNN. The rule-set can be viewed as a normal logic program wherein each predicate's truth value depends on a kernel group in the CNN. Each predicate in the rule-set is mapped to a concept using a few semantic segmentation masks of the images used for training, to make it human-understandable. The last layers of the CNN can then be replaced by this rule-set to obtain the NeSy-G model which can then be used for the image classification task. The goal directed ASP system s(CASP) can be used to obtain the justification of any prediction made using the NeSy-G model. We also propose a novel algorithm for labeling each predicate in the rule-set with the semantic concept(s) that its corresponding kernel group represents.