# Guaranteeing Control Requirements via Reward Shaping in Reinforcement Learning

arXiv: https://arxiv.org/abs/2311.10026 · HTML: https://mlsanity.com/paper/2311.10026/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 56 references that are in the map.
- First posted on arXiv 16 Nov 2023.

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2002.00444/) (2020)
- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [Safe Model-based Reinforcement Learning with Stability Guarantees](https://mlsanity.com/paper/1705.08551/) (2017)
- [A Lyapunov-based Approach to Safe Reinforcement Learning](https://mlsanity.com/paper/1805.07708/) (2018)
- [Learning by Playing - Solving Sparse Reward Tasks from Scratch](https://mlsanity.com/paper/1802.10567/) (2018)
- [Learning to Utilize Shaping Rewards: A New Approach of Reward Shaping](https://mlsanity.com/paper/2011.02669/) (2020)
- [Safe Reinforcement Learning Using Robust MPC](https://mlsanity.com/paper/1906.04005/) (2019)
- [Reinforcement Learning for Safety-Critical Control under Model Uncertainty, using Control Lyapunov Functions and Control Barrier Functions](https://mlsanity.com/paper/2004.07584/) (2020)
- [Unpacking Reward Shaping: Understanding the Benefits of Reward Engineering on Sample Complexity](https://mlsanity.com/paper/2210.09579/) (2022)
- [Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstration](https://mlsanity.com/paper/2202.04628/) (2022)
- [Subgoal-based Reward Shaping to Improve Efficiency in Reinforcement Learning](https://mlsanity.com/paper/2104.06411/) (2021)
- [Control-Tutored Reinforcement Learning: Towards the Integration of Data-Driven and Model-Based Control](https://mlsanity.com/paper/2112.06018/) (2021)
- [Inverse stochastic optimal controls](https://mlsanity.com/paper/2005.11485/) (2020)
- [Inverse Optimal Control with Discount Factor for Continuous and Discrete-Time Control-Affine Systems and Reinforcement Learning](https://mlsanity.com/paper/2211.09917/) (2022)
- [CT-DQN: Control-Tutored Deep Reinforcement Learning](https://mlsanity.com/paper/2212.01343/) (2022)
- [On cost design in applications of optimal control](https://mlsanity.com/paper/2104.00917/) (2021)
- [On a Probabilistic Approach for Inverse Data-Driven Optimal Control](https://mlsanity.com/paper/2303.17957/) (2023)

## Built on by (most cited first, 5 of 5)

- [Comprehensive Overview of Reward Engineering and Shaping in Advancing Reinforcement Learning Applications](https://mlsanity.com/paper/2408.10215/) (2024)
- [A Safe Reinforcement Learning Algorithm for Supervisory Control of Power Plants](https://mlsanity.com/paper/2401.13020/) (2024)
- [Toward Computationally Efficient Inverse Reinforcement Learning via Reward Shaping](https://mlsanity.com/paper/2312.09983/) (2023)
- [Data-driven architecture to encode information in the kinematics of robots and artificial avatars](https://mlsanity.com/paper/2403.06557/) (2024)
- [CMR: Contractive Mapping Embeddings for Robust Humanoid Locomotion on Unstructured Terrains](https://mlsanity.com/paper/2602.03511/) (2026)

## Abstract (arXiv)

In addressing control problems such as regulation and tracking through reinforcement learning, it is often required to guarantee that the acquired policy meets essential performance and stability criteria such as a desired settling time and steady-state error prior to deployment. Motivated by this necessity, we present a set of results and a systematic reward shaping procedure that (i) ensures the optimal policy generates trajectories that align with specified control requirements and (ii) allows to assess whether any given policy satisfies them. We validate our approach through comprehensive numerical experiments conducted in two representative environments from OpenAI Gym: the Inverted Pendulum swing-up problem and the Lunar Lander. Utilizing both tabular and deep reinforcement learning methods, our experiments consistently affirm the efficacy of our proposed framework, highlighting its effectiveness in ensuring policy adherence to the prescribed control requirements.