# Symbolic Knowledge Extraction and Injection with Sub-symbolic Predictors: A Systematic Literature Review

arXiv: https://arxiv.org/abs/2501.14836 · HTML: https://mlsanity.com/paper/2501.14836/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 61 references that are in the map.
- First posted on arXiv 23 Jan 2025.

## Builds on (most cited first, 14 of 14)

- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [Object Detection with Deep Learning: A Review](https://mlsanity.com/paper/1807.05511/) (2018)
- [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](https://mlsanity.com/paper/2103.11251/) (2021)
- [European Union regulations on algorithmic decision-making and a "right to explanation"](https://mlsanity.com/paper/1606.08813/) (2016)
- [Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systems](https://mlsanity.com/paper/1903.12394/) (2019)
- [Neural-Symbolic Learning and Reasoning: A Survey and Interpretation](https://mlsanity.com/paper/1711.03902/) (2017)
- [Graph Neural Networks Meet Neural-Symbolic Computing: A Survey and Perspective](https://mlsanity.com/paper/2003.00330/) (2020)
- [Interpretable Predictions of Tree-based Ensembles via Actionable Feature Tweaking](https://mlsanity.com/paper/1706.06691/) (2017)
- [Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Framework](https://mlsanity.com/paper/2103.02885/) (2021)
- [Embedding Symbolic Knowledge into Deep Networks](https://mlsanity.com/paper/1909.01161/) (2019)
- [Rule Extraction Algorithm for Deep Neural Networks: A Review](https://mlsanity.com/paper/1610.05267/) (2016)

## Built on by (most cited first, 7 of 7)

- [A Comparative Study of Neurosymbolic AI Approaches to Interpretable Logical Reasoning](https://mlsanity.com/paper/2508.03366/) (2025)
- [A Learnability Analysis on Neuro-Symbolic Learning](https://mlsanity.com/paper/2503.16797/) (2025)
- [Towards Responsible and Trustworthy Educational Data Mining: Comparing Symbolic, Sub-Symbolic, and Neural-Symbolic AI Methods](https://mlsanity.com/paper/2504.00615/) (2025)
- [Towards responsible AI for education: Hybrid human-AI to confront the Elephant in the room](https://mlsanity.com/paper/2504.16148/) (2025)
- [Problems With Large Language Models for Learner Modelling: Why LLMs Alone Fall Short for Responsible Tutoring in K--12 Education](https://mlsanity.com/paper/2512.23036/) (2025)
- [Integrating Machine Learning into Belief-Desire-Intention Agents: Current Advances and Open Challenges](https://mlsanity.com/paper/2510.20641/) (2025)
- [Does Reasoning Mitigate Backdoor Attacks? A Neuro-Symbolic Perspective](https://mlsanity.com/paper/2609.00464/) (2026)

## Abstract (arXiv)

In this paper we focus on the opacity issue of sub-symbolic machine learning predictors by promoting two complementary activities, namely, symbolic knowledge extraction (SKE) and injection (SKI) from and into sub-symbolic predictors. We consider as symbolic any language being intelligible and interpretable for both humans and computers. Accordingly, we propose general meta-models for both SKE and SKI, along with two taxonomies for the classification of SKE and SKI methods. By adopting an explainable artificial intelligence (XAI) perspective, we highlight how such methods can be exploited to mitigate the aforementioned opacity issue. Our taxonomies are attained by surveying and classifying existing methods from the literature, following a systematic approach, and by generalising the results of previous surveys targeting specific sub-topics of either SKE or SKI alone. More precisely, we analyse 132 methods for SKE and 117 methods for SKI, and we categorise them according to their purpose, operation, expected input/output data and predictor types. For each method, we also indicate the presence/lack of runnable software implementations. Our work may be of interest for data scientists aiming at selecting the most adequate SKE/SKI method for their needs, and also work as suggestions for researchers interested in filling the gaps of the current state of the art, as well as for developers willing to implement SKE/SKI-based technologies.