# Neurosymbolic Reasoning with Incremental Knowledge for Sample Efficient Hierarchical Reinforcement Learning

arXiv: https://arxiv.org/abs/2608.02993 · HTML: https://mlsanity.com/paper/2608.02993/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 26 of its 47 references that are in the map.
- First posted on arXiv 4 Aug 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 26 of 26)

- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [Past, Present, and Future of Simultaneous Localization And Mapping: Towards the Robust-Perception Age](https://mlsanity.com/paper/1606.05830/) (2016)
- [Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05143/) (2016)
- [Towards Continual Reinforcement Learning: A Review and Perspectives](https://mlsanity.com/paper/2012.13490/) (2020)
- [Semi-parametric Topological Memory for Navigation](https://mlsanity.com/paper/1803.00653/) (2018)
- [Learning to Navigate in Complex Environments](https://mlsanity.com/paper/1611.03673/) (2016)
- [Search on the Replay Buffer: Bridging Planning and Reinforcement Learning](https://mlsanity.com/paper/1906.05253/) (2019)
- [PDDLStream: Integrating Symbolic Planners and Blackbox Samplers via Optimistic Adaptive Planning](https://mlsanity.com/paper/1802.08705/) (2018)
- [Planning with Goal-Conditioned Policies](https://mlsanity.com/paper/1911.08453/) (2019)
- [SDRL: Interpretable and Data-efficient Deep Reinforcement Learning Leveraging Symbolic Planning](https://mlsanity.com/paper/1811.00090/) (2018)
- [PRM-RL: Long-range Robotic Navigation Tasks by Combining Reinforcement Learning and Sampling-based Planning](https://mlsanity.com/paper/1710.03937/) (2017)
- [Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?](https://mlsanity.com/paper/1909.10618/) (2019)
- [Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2110.13625/) (2021)
- [Mapping State Space using Landmarks for Universal Goal Reaching](https://mlsanity.com/paper/1908.05451/) (2019)
- [Neurosymbolic Reinforcement Learning and Planning: A Survey](https://mlsanity.com/paper/2309.01038/) (2023)
- [Leveraging Approximate Symbolic Models for Reinforcement Learning via Skill Diversity](https://mlsanity.com/paper/2202.02886/) (2022)
- [RL-RRT: Kinodynamic Motion Planning via Learning Reachability Estimators from RL Policies](https://mlsanity.com/paper/1907.04799/) (2019)
- [Learned Critical Probabilistic Roadmaps for Robotic Motion Planning](https://mlsanity.com/paper/1910.03701/) (2019)
- [Active Learning of Abstract Plan Feasibility](https://mlsanity.com/paper/2107.00683/) (2021)
- [Map Induction: Compositional spatial submap learning for efficient exploration in novel environments](https://mlsanity.com/paper/2110.12301/) (2021)
- [PDSketch: Integrated Planning Domain Programming and Learning](https://mlsanity.com/paper/2303.05501/) (2023)
- [Reconciling Spatial and Temporal Abstractions for Goal Representation](https://mlsanity.com/paper/2401.09870/) (2024)
- [Task and Motion Planning for Execution in the Real](https://mlsanity.com/paper/2406.03641/) (2024)
- [Combining Planning, Reasoning and Reinforcement Learning to solve Industrial Robot Tasks](https://mlsanity.com/paper/2212.03570/) (2022)
- [Towards an Interpretable Hierarchical Agent Framework using Semantic Goals](https://mlsanity.com/paper/2210.08412/) (2022)
- [Learning to Navigate in Mazes with Novel Layouts using Abstract Top-down Maps](https://mlsanity.com/paper/2412.12024/) (2024)

## Abstract (arXiv)

(Flat) Reinforcement Learning (RL) agents face significant challenges in environments with sparse rewards that require long-horizon reasoning. A compelling approach to improve sample efficiency is to incorporate knowledge into learning and decision-making. In standard Hierarchical RL (HRL), knowledge is encoded in a fixed, non-updatable form, such as architectural choices, and remains unchanged throughout learning. With fixed HRL, reasoning with incremental knowledge learned during exploration is impractical before sufficient environmental knowledge is acquired, leading to poor sample efficiency. In this work, we propose neurosymbolic HRL with em Incremental Knowledge (InK): symbolic high-level components perform em symbolic planning (e.g. using D^*) on an updatable representation of current InK, while low-level goal-conditioned neural modules learn motion primitives through experience using reward shaping. Experiments on navigation tasks demonstrate that incorporating InK substantially improves sample efficiency. Additionally, to perform em optimal symbolic planning given em prior knowledge about the world, we develop Belief World Tree Search. The code is available at https://github.com/CPS-research-group/inkbwts.