# Deep Explainable Relational Reinforcement Learning: A Neuro-Symbolic Approach

arXiv: https://arxiv.org/abs/2304.08349 · HTML: https://mlsanity.com/paper/2304.08349/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 61 references that are in the map.
- First posted on arXiv 17 Apr 2023.

## Builds on (most cited first, 19 of 19)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [A Brief Survey of Deep Reinforcement Learning](https://mlsanity.com/paper/1708.05866/) (2017)
- [A Semantic Loss Function for Deep Learning with Symbolic Knowledge](https://mlsanity.com/paper/1711.11157/) (2017)
- [Distilling a Neural Network Into a Soft Decision Tree](https://mlsanity.com/paper/1711.09784/) (2017)
- [Learning Explanatory Rules from Noisy Data](https://mlsanity.com/paper/1711.04574/) (2017)
- [Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents](https://mlsanity.com/paper/1802.08757/) (2018)
- [Graph Neural Networks Meet Neural-Symbolic Computing: A Survey and Perspective](https://mlsanity.com/paper/2003.00330/) (2020)
- [SDRL: Interpretable and Data-efficient Deep Reinforcement Learning Leveraging Symbolic Planning](https://mlsanity.com/paper/1811.00090/) (2018)
- [Relational Deep Reinforcement Learning](https://mlsanity.com/paper/1806.01830/) (2018)
- [Toward Interpretable Deep Reinforcement Learning with Linear Model U-Trees](https://mlsanity.com/paper/1807.05887/) (2018)
- [Neural Logic Reinforcement Learning](https://mlsanity.com/paper/1904.10729/) (2019)
- [PEORL: Integrating Symbolic Planning and Hierarchical Reinforcement Learning for Robust Decision-Making](https://mlsanity.com/paper/1804.07779/) (2018)
- [Transparency and Explanation in Deep Reinforcement Learning Neural Networks](https://mlsanity.com/paper/1809.06061/) (2018)
- [Networked Multi-Agent Reinforcement Learning with Emergent Communication](https://mlsanity.com/paper/2004.02780/) (2020)
- [Size Independent Neural Transfer for RDDL Planning](https://mlsanity.com/paper/1902.03081/) (2019)
- [Incorporating Relational Background Knowledge into Reinforcement Learning via Differentiable Inductive Logic Programming](https://mlsanity.com/paper/2003.10386/) (2020)
- [T-Norms Driven Loss Functions for Machine Learning](https://mlsanity.com/paper/1907.11468/) (2019)

## Built on by (most cited first, 15 of 15)

- [Bridging the Human-AI Knowledge Gap: Concept Discovery and Transfer in AlphaZero](https://mlsanity.com/paper/2310.16410/) (2023)
- [Neuro-Symbolic AI in 2024: A Systematic Review](https://mlsanity.com/paper/2501.05435/) (2025)
- [BlendRL: A Framework for Merging Symbolic and Neural Policy Learning](https://mlsanity.com/paper/2410.11689/) (2024)
- [Neuro-Symbolic AI: Explainability, Challenges, and Future Trends](https://mlsanity.com/paper/2411.04383/) (2024)
- [Multimodal LLM-assisted Evolutionary Search for Programmatic Control Policies](https://mlsanity.com/paper/2508.05433/) (2025)
- [Towards Safe Autonomous Driving Policies using a Neuro-Symbolic Deep Reinforcement Learning Approach](https://mlsanity.com/paper/2307.01316/) (2023)
- [Graph-Enhanced Policy Optimization in LLM Agent Training](https://mlsanity.com/paper/2510.26270/) (2025)
- [Online inductive learning from answer sets for efficient reinforcement learning exploration](https://mlsanity.com/paper/2501.07445/) (2025)
- [Deep Learning for Generalised Planning with Background Knowledge](https://mlsanity.com/paper/2410.07923/) (2024)
- [Inductive Learning of Robot Task Knowledge from Raw Data and Online Expert Feedback](https://mlsanity.com/paper/2501.07507/) (2025)
- [Vejde: A Framework for Inductive Deep Reinforcement Learning Based on Factor Graph Color Refinement](https://mlsanity.com/paper/2509.09219/) (2025)
- [Human-Allied Relational Reinforcement Learning](https://mlsanity.com/paper/2510.16188/) (2025)
- [Sample-Efficient Neurosymbolic Deep Reinforcement Learning](https://mlsanity.com/paper/2601.02850/) (2026)
- [GRAIL: Autonomous Concept Grounding for Neuro-Symbolic Reinforcement Learning](https://mlsanity.com/paper/2604.16871/) (2026)
- [Sample-efficient Neuro-symbolic Proximal Policy Optimization](https://mlsanity.com/paper/2604.25534/) (2026)

## Abstract (arXiv)

Despite numerous successes in Deep Reinforcement Learning (DRL), the learned policies are not interpretable. Moreover, since DRL does not exploit symbolic relational representations, it has difficulties in coping with structural changes in its environment (such as increasing the number of objects). Relational Reinforcement Learning, on the other hand, inherits the relational representations from symbolic planning to learn reusable policies. However, it has so far been unable to scale up and exploit the power of deep neural networks. We propose Deep Explainable Relational Reinforcement Learning (DERRL), a framework that exploits the best of both -- neural and symbolic worlds. By resorting to a neuro-symbolic approach, DERRL combines relational representations and constraints from symbolic planning with deep learning to extract interpretable policies. These policies are in the form of logical rules that explain how each decision (or action) is arrived at. Through several experiments, in setups like the Countdown Game, Blocks World, Gridworld, and Traffic, we show that the policies learned by DERRL can be applied to different configurations and contexts, hence generalizing to environmental modifications.