# Towards Deep Symbolic Reinforcement Learning

arXiv: https://arxiv.org/abs/1609.05518 · HTML: https://mlsanity.com/paper/1609.05518/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Sep 2016.

## Built on by (most cited first, 30 of 42)

- [End-to-end symbolic regression with transformers](https://mlsanity.com/paper/2204.10532/) (2022)
- [Acquisition of Chess Knowledge in AlphaZero](https://mlsanity.com/paper/2111.09259/) (2021)
- [A Survey on Interpretable Reinforcement Learning](https://mlsanity.com/paper/2112.13112/) (2021)
- [An Investigation into Pre-Training Object-Centric Representations for Reinforcement Learning](https://mlsanity.com/paper/2302.04419/) (2023)
- [Can We Leave Deepfake Data Behind in Training Deepfake Detector?](https://mlsanity.com/paper/2408.17052/) (2024)
- [Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning](https://mlsanity.com/paper/2107.12544/) (2021)
- [Bridging the Human-AI Knowledge Gap: Concept Discovery and Transfer in AlphaZero](https://mlsanity.com/paper/2310.16410/) (2023)
- [Modular Design Patterns for Hybrid Learning and Reasoning Systems: a taxonomy, patterns and use cases](https://mlsanity.com/paper/2102.11965/) (2021)
- [A Survey on Neural-symbolic Learning Systems](https://mlsanity.com/paper/2111.08164/) (2021)
- [Towards Data-and Knowledge-Driven Artificial Intelligence: A Survey on Neuro-Symbolic Computing](https://mlsanity.com/paper/2210.15889/) (2022)
- [Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoning](https://mlsanity.com/paper/2207.09081/) (2022)
- [Symmetry-Based Representations for Artificial and Biological General Intelligence](https://mlsanity.com/paper/2203.09250/) (2022)
- [Neurosymbolic Reinforcement Learning and Planning: A Survey](https://mlsanity.com/paper/2309.01038/) (2023)
- [Unsupervised Object-Based Transition Models for 3D Partially Observable Environments](https://mlsanity.com/paper/2103.04693/) (2021)
- [Compositional Multi-Object Reinforcement Learning with Linear Relation Networks](https://mlsanity.com/paper/2201.13388/) (2022)
- [Abstraction for Deep Reinforcement Learning](https://mlsanity.com/paper/2202.05839/) (2022)
- [Structure in Deep Reinforcement Learning: A Survey and Open Problems](https://mlsanity.com/paper/2306.16021/) (2023)
- [Stacking Brick by Brick: Aligned Feature Isolation for Incremental Face Forgery Detection](https://mlsanity.com/paper/2411.11396/) (2024)
- [Neuro-Symbolic Constraint Programming for Structured Prediction](https://mlsanity.com/paper/2103.17232/) (2021)
- [Neuro-Symbolic AI: Explainability, Challenges, and Future Trends](https://mlsanity.com/paper/2411.04383/) (2024)
- [Online Learning of Reusable Abstract Models for Object Goal Navigation](https://mlsanity.com/paper/2203.02583/) (2022)
- [Symbolic Visual Reinforcement Learning: A Scalable Framework with Object-Level Abstraction and Differentiable Expression Search](https://mlsanity.com/paper/2212.14849/) (2022)
- [Goal Space Abstraction in Hierarchical Reinforcement Learning via Set-Based Reachability Analysis](https://mlsanity.com/paper/2309.07675/) (2023)
- [Learning Multi-Object Positional Relationships via Emergent Communication](https://mlsanity.com/paper/2302.08084/) (2023)
- [Online Grounding of Symbolic Planning Domains in Unknown Environments](https://mlsanity.com/paper/2112.10007/) (2021)
- [A Review of Symbolic, Subsymbolic and Hybrid Methods for Sequential Decision Making](https://mlsanity.com/paper/2304.10590/) (2023)
- [ADAPTER-RL: Adaptation of Any Agent using Reinforcement Learning](https://mlsanity.com/paper/2311.11537/) (2023)
- [A Synergistic Approach In Network Intrusion Detection By Neurosymbolic AI](https://mlsanity.com/paper/2406.00938/) (2024)
- [ExReg: Wide-range Photo Exposure Correction via a Multi-dimensional Regressor with Attention](https://mlsanity.com/paper/2212.14801/) (2022)
- [Teach Me How to Learn: A Perspective Review towards User-centered Neuro-symbolic Learning for Robotic Surgical Systems](https://mlsanity.com/paper/2307.03853/) (2023)

## Abstract (arXiv)

Deep reinforcement learning (DRL) brings the power of deep neural networks to bear on the generic task of trial-and-error learning, and its effectiveness has been convincingly demonstrated on tasks such as Atari video games and the game of Go. However, contemporary DRL systems inherit a number of shortcomings from the current generation of deep learning techniques. For example, they require very large datasets to work effectively, entailing that they are slow to learn even when such datasets are available. Moreover, they lack the ability to reason on an abstract level, which makes it difficult to implement high-level cognitive functions such as transfer learning, analogical reasoning, and hypothesis-based reasoning. Finally, their operation is largely opaque to humans, rendering them unsuitable for domains in which verifiability is important. In this paper, we propose an end-to-end reinforcement learning architecture comprising a neural back end and a symbolic front end with the potential to overcome each of these shortcomings. As proof-of-concept, we present a preliminary implementation of the architecture and apply it to several variants of a simple video game. We show that the resulting system -- though just a prototype -- learns effectively, and, by acquiring a set of symbolic rules that are easily comprehensible to humans, dramatically outperforms a conventional, fully neural DRL system on a stochastic variant of the game.