# Knowledge-Guided Exploration in Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2210.15670 · HTML: https://mlsanity.com/paper/2210.15670/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 42 references that are in the map.
- First posted on arXiv 26 Oct 2022.

## Builds on (most cited first, 24 of 24)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [Learning to reinforcement learn](https://mlsanity.com/paper/1611.05763/) (2016)
- [Safe Model-based Reinforcement Learning with Stability Guarantees](https://mlsanity.com/paper/1705.08551/) (2017)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning](https://mlsanity.com/paper/1708.02596/) (2017)
- [Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation](https://mlsanity.com/paper/1604.06057/) (2016)
- [Safe Exploration in Continuous Action Spaces](https://mlsanity.com/paper/1801.08757/) (2018)
- [(More) Efficient Reinforcement Learning via Posterior Sampling](https://mlsanity.com/paper/1306.0940/) (2013)
- [Deep Reinforcement Learning framework for Autonomous Driving](https://mlsanity.com/paper/1704.02532/) (2017)
- [Model-Based Reinforcement Learning via Meta-Policy Optimization](https://mlsanity.com/paper/1809.05214/) (2018)
- [Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning](https://mlsanity.com/paper/1809.02121/) (2018)
- [Investigating Human Priors for Playing Video Games](https://mlsanity.com/paper/1802.10217/) (2018)
- [BBQ-Networks: Efficient Exploration in Deep Reinforcement Learning for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1711.05715/) (2017)
- [End-to-End Deep Reinforcement Learning for Lane Keeping Assist](https://mlsanity.com/paper/1612.04340/) (2016)
- [What can you do with a rock? Affordance extraction via word embeddings](https://mlsanity.com/paper/1703.03429/) (2017)
- [Simulated Car Racing Championship: Competition Software Manual](https://mlsanity.com/paper/1304.1672/) (2013)
- [Adaptive Exploration-Exploitation Tradeoff for Opportunistic Bandits](https://mlsanity.com/paper/1709.04004/) (2017)
- [Concurrent learning-based approximate optimal regulation](https://mlsanity.com/paper/1304.3477/) (2013)
- [Combating Reinforcement Learning's Sisyphean Curse with Intrinsic Fear](https://mlsanity.com/paper/1611.01211/) (2016)

## Built on by (most cited first, 2 of 2)

- [Improving Sample Efficiency of Reinforcement Learning with Background Knowledge from Large Language Models](https://mlsanity.com/paper/2407.03964/) (2024)
- [AUTONODE: A Neuro-Graphic Self-Learnable Engine for Cognitive GUI Automation](https://mlsanity.com/paper/2403.10171/) (2024)

## Abstract (arXiv)

This paper proposes a new method to drastically speed up deep reinforcement learning (deep RL) training for problems that have the property of state-action permissibility (SAP). Two types of permissibility are defined under SAP. The first type says that after an action at is performed in a state st and the agent has reached the new state st+1, the agent can decide whether at is permissible or not permissible in st. The second type says that even without performing at in st, the agent can already decide whether at is permissible or not in st. An action is not permissible in a state if the action can never lead to an optimal solution and thus should not be tried (over and over again). We incorporate the proposed SAP property and encode action permissibility knowledge into two state-of-the-art deep RL algorithms to guide their state-action exploration together with a virtual stopping strategy. Results show that the SAP-based guidance can markedly speed up RL training.