# Synthesising Reinforcement Learning Policies through Set-Valued Inductive Rule Learning

arXiv: https://arxiv.org/abs/2106.06009 · HTML: https://mlsanity.com/paper/2106.06009/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 36 references that are in the map.
- First posted on arXiv 10 Jun 2021.

## Builds on (most cited first, 7 of 7)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Policy Distillation](https://mlsanity.com/paper/1511.06295/) (2015)
- [Distilling a Neural Network Into a Soft Decision Tree](https://mlsanity.com/paper/1711.09784/) (2017)
- [Explainable Reinforcement Learning Through a Causal Lens](https://mlsanity.com/paper/1905.10958/) (2019)
- [Deep reinforcement learning for large-scale epidemic control](https://mlsanity.com/paper/2003.13676/) (2020)
- [Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics](https://mlsanity.com/paper/1903.04193/) (2019)

## Built on by (most cited first, 2 of 2)

- [Counterfactual Explainer Framework for Deep Reinforcement Learning Models Using Policy Distillation](https://mlsanity.com/paper/2305.16532/) (2023)
- [From Black Box to Executable Logic: Explainable Reinforcement Learning through Prolog Expert Systems](https://mlsanity.com/paper/2607.15459/) (2026)

## Abstract (arXiv)

Today's advanced Reinforcement Learning algorithms produce black-box policies, that are often difficult to interpret and trust for a person. We introduce a policy distilling algorithm, building on the CN2 rule mining algorithm, that distills the policy into a rule-based decision system. At the core of our approach is the fact that an RL process does not just learn a policy, a mapping from states to actions, but also produces extra meta-information, such as action values indicating the quality of alternative actions. This meta-information can indicate whether more than one action is near-optimal for a certain state. We extend CN2 to make it able to leverage knowledge about equally-good actions to distill the policy into fewer rules, increasing its interpretability by a person. Then, to ensure that the rules explain a valid, non-degenerate policy, we introduce a refinement algorithm that fine-tunes the rules to obtain good performance when executed in the environment. We demonstrate the applicability of our algorithm on the Mario AI benchmark, a complex task that requires modern reinforcement learning algorithms including neural networks. The explanations we produce capture the learned policy in only a few rules, that allow a person to understand what the black-box agent learned. Source code: https://gitlab.ai.vub.ac.be/yocoppen/svcn2