# An Empirical Investigation into Deep and Shallow Rule Learning

arXiv: https://arxiv.org/abs/2106.10254 · HTML: https://mlsanity.com/paper/2106.10254/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 74 references that are in the map.
- First posted on arXiv 18 Jun 2021.

## Builds on (most cited first, 13 of 13)

- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [Learning Explanatory Rules from Noisy Data](https://mlsanity.com/paper/1711.04574/) (2017)
- [Trained Ternary Quantization](https://mlsanity.com/paper/1612.01064/) (2016)
- [Ternary Weight Networks](https://mlsanity.com/paper/1605.04711/) (2016)
- [Binary Neural Networks: A Survey](https://mlsanity.com/paper/2004.03333/) (2020)
- [Local Rule-Based Explanations of Black Box Decision Systems](https://mlsanity.com/paper/1805.10820/) (2018)
- [GLocalX -- From Local to Global Explanations of Black Box AI Models](https://mlsanity.com/paper/2101.07685/) (2021)
- [Classifier Chains: A Review and Perspectives](https://mlsanity.com/paper/1912.13405/) (2019)
- [Multi-Target Prediction: A Unifying View on Problems and Methods](https://mlsanity.com/paper/1809.02352/) (2018)
- [Learning Gradient Boosted Multi-label Classification Rules](https://mlsanity.com/paper/2006.13346/) (2020)
- [Learning Structured Declarative Rule Sets -- A Challenge for Deep Discrete Learning](https://mlsanity.com/paper/2012.04377/) (2020)
- [Multi-label Classification using Labels as Hidden Nodes](https://mlsanity.com/paper/1503.09022/) (2015)

## Built on by (most cited first, 5 of 5)

- [A framework for comprehensible multi-modal detection of cyber threats](https://mlsanity.com/paper/2111.05764/) (2021)
- [Bayes Point Rule Set Learning](https://mlsanity.com/paper/2204.05251/) (2022)
- [HorNets: Learning from Discrete and Continuous Signals with Routing Neural Networks](https://mlsanity.com/paper/2501.14346/) (2025)
- [(Sometimes) Less is More: Mitigating the Complexity of Rule-based Representation for Interpretable Classification](https://mlsanity.com/paper/2509.22384/) (2025)
- [Optimizing Sparse Boolean Rule Networks using Mixed Integer Linear Programming](https://mlsanity.com/paper/2610.20170/) (2026)

## Abstract (arXiv)

Inductive rule learning is arguably among the most traditional paradigms in machine learning. Although we have seen considerable progress over the years in learning rule-based theories, all state-of-the-art learners still learn descriptions that directly relate the input features to the target concept. In the simplest case, concept learning, this is a disjunctive normal form (DNF) description of the positive class. While it is clear that this is sufficient from a logical point of view because every logical expression can be reduced to an equivalent DNF expression, it could nevertheless be the case that more structured representations, which form deep theories by forming intermediate concepts, could be easier to learn, in very much the same way as deep neural networks are able to outperform shallow networks, even though the latter are also universal function approximators. In this paper, we empirically compare deep and shallow rule learning with a uniform general algorithm, which relies on greedy mini-batch based optimization. Our experiments on both artificial and real-world benchmark data indicate that deep rule networks outperform shallow networks.