# Handling Sparse Rewards in Reinforcement Learning Using Model Predictive Control

arXiv: https://arxiv.org/abs/2210.01525 · HTML: https://mlsanity.com/paper/2210.01525/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 37 references that are in the map.
- First posted on arXiv 4 Oct 2022.

## Builds on (most cited first, 22 of 22)

- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards](https://mlsanity.com/paper/1707.08817/) (2017)
- [Residual Policy Learning](https://mlsanity.com/paper/1812.06298/) (2018)
- [acados: a modular open-source framework for fast embedded optimal control](https://mlsanity.com/paper/1910.13753/) (2019)
- [Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstration](https://mlsanity.com/paper/2202.04628/) (2022)
- [Learning Deep Control Policies for Autonomous Aerial Vehicles with MPC-Guided Policy Search](https://mlsanity.com/paper/1509.06791/) (2015)
- [Reinforced Imitation: Sample Efficient Deep Reinforcement Learning for Map-less Navigation by Leveraging Prior Demonstrations](https://mlsanity.com/paper/1805.07095/) (2018)
- [Deep Q-learning from Demonstrations](https://mlsanity.com/paper/1704.03732/) (2017)
- [Where to go next: Learning a Subgoal Recommendation Policy for Navigation Among Pedestrians](https://mlsanity.com/paper/2102.13073/) (2021)
- [Improved Deep Reinforcement Learning with Expert Demonstrations for Urban Autonomous Driving](https://mlsanity.com/paper/2102.09243/) (2021)
- [Learning with Training Wheels: Speeding up Training with a Simple Controller for Deep Reinforcement Learning](https://mlsanity.com/paper/1812.05027/) (2018)
- [Integrating Behavior Cloning and Reinforcement Learning for Improved Performance in Dense and Sparse Reward Environments](https://mlsanity.com/paper/1910.04281/) (2019)
- [Safe end-to-end imitation learning for model predictive control](https://mlsanity.com/paper/1803.10231/) (2018)
- [Learning of Long-Horizon Sparse-Reward Robotic Manipulator Tasks with Base Controllers](https://mlsanity.com/paper/2011.12105/) (2020)
- [Infusing model predictive control into meta-reinforcement learning for mobile robots in dynamic environments](https://mlsanity.com/paper/2109.07120/) (2021)
- [Goal-constrained Sparse Reinforcement Learning for End-to-End Driving](https://mlsanity.com/paper/2103.09189/) (2021)
- [Learning Personalized Human-Aware Robot Navigation Using Virtual Reality Demonstrations from a User Study](https://mlsanity.com/paper/2203.14741/) (2022)
- [An Efficient Real-Time NMPC for Quadrotor Position Control under Communication Time-Delay](https://mlsanity.com/paper/2010.11264/) (2020)
- [End-Effector Stabilization of a 10-DOF Mobile Manipulator using Nonlinear Model Predictive Control](https://mlsanity.com/paper/2103.13153/) (2021)

## Built on by (most cited first, 5 of 5)

- [Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classification](https://mlsanity.com/paper/2502.02133/) (2025)
- [ETGL-DDPG: A Deep Deterministic Policy Gradient Algorithm for Sparse Reward Continuous Control](https://mlsanity.com/paper/2410.05225/) (2024)
- [MPC-Injection: Biasing Off-Policy Locomotion RL Toward Controller-Induced Behavior Basins](https://mlsanity.com/paper/2606.26392/) (2026)
- [Towards a Reward-Free Reinforcement Learning Framework for Vehicle Control](https://mlsanity.com/paper/2502.15262/) (2025)
- [Bidirectional Task-Motion Planning Based on Hierarchical Reinforcement Learning for Strategic Confrontation](https://mlsanity.com/paper/2504.15876/) (2025)

## Abstract (arXiv)

Reinforcement learning (RL) has recently proven great success in various domains. Yet, the design of the reward function requires detailed domain expertise and tedious fine-tuning to ensure that agents are able to learn the desired behaviour. Using a sparse reward conveniently mitigates these challenges. However, the sparse reward represents a challenge on its own, often resulting in unsuccessful training of the agent. In this paper, we therefore address the sparse reward problem in RL. Our goal is to find an effective alternative to reward shaping, without using costly human demonstrations, that would also be applicable to a wide range of domains. Hence, we propose to use model predictive control~(MPC) as an experience source for training RL agents in sparse reward environments. Without the need for reward shaping, we successfully apply our approach in the field of mobile robot navigation both in simulation and real-world experiments with a Kuboki Turtlebot 2. We furthermore demonstrate great improvement over pure RL algorithms in terms of success rate as well as number of collisions and timeouts. Our experiments show that MPC as an experience source improves the agent's learning process for a given task in the case of sparse rewards.