# Neuro-Symbolic Reinforcement Learning with First-Order Logic

arXiv: https://arxiv.org/abs/2110.10963 · HTML: https://mlsanity.com/paper/2110.10963/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 21 references that are in the map.
- First posted on arXiv 21 Oct 2021.

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [TextWorld: A Learning Environment for Text-based Games](https://mlsanity.com/paper/1806.11532/) (2018)
- [Neural Logic Machines](https://mlsanity.com/paper/1904.11694/) (2019)
- [Logical Neural Networks](https://mlsanity.com/paper/2006.13155/) (2020)
- [Language Understanding for Text-based Games Using Deep Reinforcement Learning](https://mlsanity.com/paper/1506.08941/) (2015)
- [Playing Text-Adventure Games with Graph-Based Deep Reinforcement Learning](https://mlsanity.com/paper/1812.01628/) (2018)
- [Learning Dynamic Belief Graphs to Generalize on Text-Based Games](https://mlsanity.com/paper/2002.09127/) (2020)
- [Text-based RL Agents with Commonsense Knowledge: New Challenges, Environments and Baselines](https://mlsanity.com/paper/2010.03790/) (2020)
- [Counting to Explore and Generalize in Text-based Games](https://mlsanity.com/paper/1806.11525/) (2018)
- [LeDeepChef: Deep Reinforcement Learning Agent for Families of Text-Based Games](https://mlsanity.com/paper/1909.01646/) (2019)
- [Internal Model from Observations for Reward Shaping](https://mlsanity.com/paper/1806.01267/) (2018)
- [Reinforcement Learning with External Knowledge by using Logical Neural Networks](https://mlsanity.com/paper/2103.02363/) (2021)
- [Bootstrapped Q-learning with Context Relevant Observation Pruning to Generalize in Text-based Games](https://mlsanity.com/paper/2009.11896/) (2020)
- [DAQN: Deep Auto-encoder and Q-Network](https://mlsanity.com/paper/1806.00630/) (2018)

## Built on by (most cited first, 30 of 30)

- [Interpretable and Explainable Logical Policies via Neurally Guided Symbolic Abstraction](https://mlsanity.com/paper/2306.01439/) (2023)
- [Neuro-Symbolic AI in 2024: A Systematic Review](https://mlsanity.com/paper/2501.05435/) (2025)
- [Introspective Tips: Large Language Model for In-Context Decision Making](https://mlsanity.com/paper/2305.11598/) (2023)
- [Neurosymbolic Reinforcement Learning and Planning: A Survey](https://mlsanity.com/paper/2309.01038/) (2023)
- [Interpretable Concept Bottlenecks to Align Reinforcement Learning Agents](https://mlsanity.com/paper/2401.05821/) (2024)
- [BlendRL: A Framework for Merging Symbolic and Neural Policy Learning](https://mlsanity.com/paper/2410.11689/) (2024)
- [TextWorldExpress: Simulating Text Games at One Million Steps Per Second](https://mlsanity.com/paper/2208.01174/) (2022)
- [Learning to Follow Instructions in Text-Based Games](https://mlsanity.com/paper/2211.04591/) (2022)
- [HackAtari: Atari Learning Environments for Robust and Continual Reinforcement Learning](https://mlsanity.com/paper/2406.03997/) (2024)
- [Behavior Cloned Transformers are Neurosymbolic Reasoners](https://mlsanity.com/paper/2210.07382/) (2022)
- [Neuro-Symbolic AI: Explainability, Challenges, and Future Trends](https://mlsanity.com/paper/2411.04383/) (2024)
- [Symbolic Visual Reinforcement Learning: A Scalable Framework with Object-Level Abstraction and Differentiable Expression Search](https://mlsanity.com/paper/2212.14849/) (2022)
- [LogiCity: Advancing Neuro-Symbolic AI with Abstract Urban Simulation](https://mlsanity.com/paper/2411.00773/) (2024)
- [Towards Safe Autonomous Driving Policies using a Neuro-Symbolic Deep Reinforcement Learning Approach](https://mlsanity.com/paper/2307.01316/) (2023)
- [Temporal Knowledge-Graph Memory in a Partially Observable Environment](https://mlsanity.com/paper/2408.05861/) (2024)
- [Learning Symbolic Rules over Abstract Meaning Representations for Textual Reinforcement Learning](https://mlsanity.com/paper/2307.02689/) (2023)
- [Neural DNF-MT: A Neuro-symbolic Approach for Learning Interpretable and Editable Policies](https://mlsanity.com/paper/2501.03888/) (2025)
- [EXPLORER: Exploration-guided Reasoning for Textual Reinforcement Learning](https://mlsanity.com/paper/2403.10692/) (2024)
- [SMOSE: Sparse Mixture of Shallow Experts for Interpretable Reinforcement Learning in Continuous Control Tasks](https://mlsanity.com/paper/2412.13053/) (2024)
- [A Synergistic Approach In Network Intrusion Detection By Neurosymbolic AI](https://mlsanity.com/paper/2406.00938/) (2024)
- [Neural Reasoning Networks: Efficient Interpretable Neural Networks With Automatic Textual Explanations](https://mlsanity.com/paper/2410.07966/) (2024)
- [Commonsense Knowledge from Scene Graphs for Textual Environments](https://mlsanity.com/paper/2210.14162/) (2022)
- [Symbolic Imitation Learning: From Black-Box to Explainable Driving Policies](https://mlsanity.com/paper/2309.16025/) (2023)
- [Efficient Open-world Reinforcement Learning via Knowledge Distillation and Autonomous Rule Discovery](https://mlsanity.com/paper/2311.14270/) (2023)
- [Three Pathways to Neurosymbolic Reinforcement Learning with Interpretable Model and Policy Networks](https://mlsanity.com/paper/2402.05307/) (2024)
- [DiffG-RL: Leveraging Difference between State and Common Sense](https://mlsanity.com/paper/2211.16002/) (2022)
- [Bridging the Gap: Representation Spaces in Neuro-Symbolic AI](https://mlsanity.com/paper/2411.04393/) (2024)
- [Hierarchical Reinforcement Learning with Targeted Causal Interventions](https://mlsanity.com/paper/2507.04373/) (2025)
- [Disentangling Neural Disjunctive Normal Form Models](https://mlsanity.com/paper/2507.10546/) (2025)
- [GRAIL: Autonomous Concept Grounding for Neuro-Symbolic Reinforcement Learning](https://mlsanity.com/paper/2604.16871/) (2026)

## Abstract (arXiv)

Deep reinforcement learning (RL) methods often require many trials before convergence, and no direct interpretability of trained policies is provided. In order to achieve fast convergence and interpretability for the policy in RL, we propose a novel RL method for text-based games with a recent neuro-symbolic framework called Logical Neural Network, which can learn symbolic and interpretable rules in their differentiable network. The method is first to extract first-order logical facts from text observation and external word meaning network (ConceptNet), then train a policy in the network with directly interpretable logical operators. Our experimental results show RL training with the proposed method converges significantly faster than other state-of-the-art neuro-symbolic methods in a TextWorld benchmark.