# MPC-based Reinforcement Learning for Economic Problems with Application to Battery Storage

arXiv: https://arxiv.org/abs/2104.02411 · HTML: https://mlsanity.com/paper/2104.02411/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 18 references that are in the map.
- First posted on arXiv 6 Apr 2021.

## Builds on (most cited first, 5 of 5)

- [Learning-based Model Predictive Control for Safe Exploration](https://mlsanity.com/paper/1803.08287/) (2018)
- [Data-driven Economic NMPC using Reinforcement Learning](https://mlsanity.com/paper/1904.04152/) (2019)
- [Reinforcement Learning for Mixed-Integer Problems Based on MPC](https://mlsanity.com/paper/2004.01430/) (2020)
- [Reinforcement Learning based on Scenario-tree MPC for ASVs](https://mlsanity.com/paper/2103.11949/) (2021)
- [Reinforcement Learning based on MPC/MHE for Unmodeled and Partially Observable Dynamics](https://mlsanity.com/paper/2103.11871/) (2021)

## Built on by (most cited first, 8 of 8)

- [Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policies](https://mlsanity.com/paper/2111.02552/) (2021)
- [UAV Path Planning Employing MPC- Reinforcement Learning Method Considering Collision Avoidance](https://mlsanity.com/paper/2302.10669/) (2023)
- [Differentiable Nonlinear Model Predictive Control](https://mlsanity.com/paper/2505.01353/) (2025)
- [Learning-based MPC from Big Data Using Reinforcement Learning](https://mlsanity.com/paper/2301.01667/) (2023)
- [MPC4RL -- A Software Package for Reinforcement Learning based on Model Predictive Control](https://mlsanity.com/paper/2501.15897/) (2025)
- [Mission-Aligned Learning-Informed Control of Autonomous Systems: Formulation and Foundations](https://mlsanity.com/paper/2507.04356/) (2025)
- [Model-free Reinforcement Learning for Model-based Control: Towards Safe, Interpretable and Sample-efficient Agents](https://mlsanity.com/paper/2507.13491/) (2025)
- [Accelerating Reinforcement Learning via MPC Solver-Gradient Guidance for Weights-varying MPC](https://mlsanity.com/paper/2609.01061/) (2026)

## Abstract (arXiv)

In this paper, we are interested in optimal control problems with purely economic costs, which often yield optimal policies having a (nearly) bang-bang structure. We focus on policy approximations based on Model Predictive Control (MPC) and the use of the deterministic policy gradient method to optimize the MPC closed-loop performance in the presence of unmodelled stochasticity or model error. When the policy has a (nearly) bang-bang structure, we observe that the policy gradient method can struggle to produce meaningful steps in the policy parameters. To tackle this issue, we propose a homotopy strategy based on the interior-point method, providing a relaxation of the policy during the learning. We investigate a specific well-known battery storage problem, and show that the proposed method delivers a homogeneous and faster learning than a classical policy gradient approach.