# Computationally Efficient Reinforcement Learning: Targeted Exploration leveraging Simple Rules

arXiv: https://arxiv.org/abs/2211.16691 · HTML: https://mlsanity.com/paper/2211.16691/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 32 references that are in the map.
- First posted on arXiv 30 Nov 2022.

## Builds on (most cited first, 15 of 15)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [Safe Exploration in Continuous Action Spaces](https://mlsanity.com/paper/1801.08757/) (2018)
- [A Review of Safe Reinforcement Learning: Methods, Theory and Applications](https://mlsanity.com/paper/2205.10330/) (2022)
- [Learning to Utilize Shaping Rewards: A New Approach of Reward Shaping](https://mlsanity.com/paper/2011.02669/) (2020)
- [Pretraining Representations for Data-Efficient Reinforcement Learning](https://mlsanity.com/paper/2106.04799/) (2021)
- [Safe Reinforcement Learning via Projection on a Safe Set: How to Achieve Optimality?](https://mlsanity.com/paper/2004.00915/) (2020)
- [Enforcing Policy Feasibility Constraints through Differentiable Projection for Energy Optimization](https://mlsanity.com/paper/2105.08881/) (2021)
- [Physically Consistent Neural Networks for building thermal modeling: theory and analysis](https://mlsanity.com/paper/2112.03212/) (2021)
- [KoGuN: Accelerating Deep Reinforcement Learning via Integrating Human Suboptimal Knowledge](https://mlsanity.com/paper/2002.07418/) (2020)
- [Near-optimal Deep Reinforcement Learning Policies from Data for Zone Temperature Control](https://mlsanity.com/paper/2203.05434/) (2022)

## Abstract (arXiv)

Model-free Reinforcement Learning (RL) generally suffers from poor sample complexity, mostly due to the need to exhaustively explore the state-action space to find well-performing policies. On the other hand, we postulate that expert knowledge of the system often allows us to design simple rules we expect good policies to follow at all times. In this work, we hence propose a simple yet effective modification of continuous actor-critic frameworks to incorporate such rules and avoid regions of the state-action space that are known to be suboptimal, thereby significantly accelerating the convergence of RL agents. Concretely, we saturate the actions chosen by the agent if they do not comply with our intuition and, critically, modify the gradient update step of the policy to ensure the learning process is not affected by the saturation step. On a room temperature control case study, it allows agents to converge to well-performing policies up to 6-7x faster than classical agents without computational overhead and while retaining good final performance.