# Rule Extraction Algorithm for Deep Neural Networks: A Review

arXiv: https://arxiv.org/abs/1610.05267 · HTML: https://mlsanity.com/paper/1610.05267/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Sep 2016.

## Built on by (most cited first, 22 of 22)

- [A Comprehensive Taxonomy for Explainable Artificial Intelligence: A Systematic Survey of Surveys on Methods and Concepts](https://mlsanity.com/paper/2105.07190/) (2021)
- [Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety](https://mlsanity.com/paper/2104.14235/) (2021)
- [Concept Embedding Analysis: A Review](https://mlsanity.com/paper/2203.13909/) (2022)
- [Efficient Decompositional Rule Extraction for Deep Neural Networks](https://mlsanity.com/paper/2111.12628/) (2021)
- [Visualizing Global Explanations of Point Cloud DNNs](https://mlsanity.com/paper/2203.09505/) (2022)
- [Local Concept Embeddings for Analysis of Concept Distributions in Vision DNN Feature Spaces](https://mlsanity.com/paper/2311.14435/) (2023)
- [Towards Symbolic XAI -- Explanation Through Human Understandable Logical Relationships Between Features](https://mlsanity.com/paper/2408.17198/) (2024)
- [Symbolic Knowledge Extraction and Injection with Sub-symbolic Predictors: A Systematic Literature Review](https://mlsanity.com/paper/2501.14836/) (2025)
- [Efficient Learning of Interpretable Classification Rules](https://mlsanity.com/paper/2205.06936/) (2022)
- [Synthesizing Context-free Grammars from Recurrent Neural Networks (Extended Version)](https://mlsanity.com/paper/2101.08200/) (2021)
- [Rule-Extraction Methods From Feedforward Neural Networks: A Systematic Literature Review](https://mlsanity.com/paper/2312.12878/) (2023)
- [DAM: Diffusion Activation Maximization for 3D Global Explanations](https://mlsanity.com/paper/2401.14938/) (2024)
- [Explainable AI the Latest Advancements and New Trends](https://mlsanity.com/paper/2505.07005/) (2025)
- [Checking extracted rules in Neural Networks](https://mlsanity.com/paper/2509.16547/) (2025)
- [Forward Composition Propagation for Explainable Neural Reasoning](https://mlsanity.com/paper/2112.12717/) (2021)
- [Exploring layerwise decision making in DNNs](https://mlsanity.com/paper/2202.00345/) (2022)
- [Automating the Design and Development of Gradient Descent Trained Expert System Networks](https://mlsanity.com/paper/2207.02845/) (2022)
- [What Do End-Users Really Want? Investigation of Human-Centered XAI for Mobile Health Apps](https://mlsanity.com/paper/2210.03506/) (2022)
- [Eclectic Rule Extraction for Explainability of Deep Neural Network based Intrusion Detection Systems](https://mlsanity.com/paper/2401.10207/) (2024)
- [Enabling Regional Explainability by Automatic and Model-agnostic Rule Extraction](https://mlsanity.com/paper/2406.17885/) (2024)
- [Your Model Is Unfair, Are You Even Aware? Inverse Relationship Between Comprehension and Trust in Explainability Visualizations of Biased ML Models](https://mlsanity.com/paper/2508.00140/) (2025)
- [Probabilistic Abstract Interpretation on Neural Networks via Grids Approximation](https://mlsanity.com/paper/2603.25266/) (2026)

## Abstract (arXiv)

Despite the highest classification accuracy in wide varieties of application areas, artificial neural network has one disadvantage. The way this Network comes to a decision is not easily comprehensible. The lack of explanation ability reduces the acceptability of neural network in data mining and decision system. This drawback is the reason why researchers have proposed many rule extraction algorithms to solve the problem. Recently, Deep Neural Network (DNN) is achieving a profound result over the standard neural network for classification and recognition problems. It is a hot machine learning area proven both useful and innovative. This paper has thoroughly reviewed various rule extraction algorithms, considering the classification scheme: decompositional, pedagogical, and eclectics. It also presents the evaluation of these algorithms based on the neural network structure with which the algorithm is intended to work. The main contribution of this review is to show that there is a limited study of rule extraction algorithm from DNN.