# Should we use model-free or model-based control? A case study of battery management systems

arXiv: https://arxiv.org/abs/2407.15313 · HTML: https://mlsanity.com/paper/2407.15313/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 10 of its 15 references that are in the map.
- First posted on arXiv 22 Jul 2024.

## Placed in

- Deep reinforcement learning methods / [RL for energy, networks and operations](https://mlsanity.com/deeprl/rl-for-energy-networks-and-operations/)

## Builds on (most cited first, 10 of 10)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Control](https://mlsanity.com/paper/1706.06491/) (2017)
- [Comparison of Deep Reinforcement Learning and Model Predictive Control for Adaptive Cruise Control](https://mlsanity.com/paper/1910.12047/) (2019)
- [Stability Constrained Reinforcement Learning for Decentralized Real-Time Voltage Control](https://mlsanity.com/paper/2209.07669/) (2022)
- [Reinforcement Learning Based Optimal Battery Control Under Cycle-based Degradation Cost](https://mlsanity.com/paper/2108.02374/) (2021)
- [SAVER: Safe Learning-Based Controller for Real-Time Voltage Regulation](https://mlsanity.com/paper/2111.15152/) (2021)
- [Energy Storage Management via Deep Q-Networks](https://mlsanity.com/paper/1903.11107/) (2019)

## Abstract (arXiv)

Reinforcement learning (RL) and model predictive control (MPC) each offer distinct advantages and limitations when applied to control problems in power and energy systems. Despite various studies on these methods, benchmarks remain lacking and the preference for RL over traditional controls is not well understood. In this work, we put forth a comparative analysis using RL- and MPC-based controllers for optimizing a battery management system (BMS). The BMS problem aims to minimize costs while adhering to operational limits. by adjusting the battery (dis)charging in response to fluctuating electricity prices over a time horizon. The MPC controller uses a learningbased forecast of future demand and price changes to formulate a multi-period linear program, that can be solved using off-the-shelf solvers. Meanwhile, the RL controller requires no timeseries modeling but instead is trained from the sample trajectories using the proximal policy optimization (PPO) algorithm. Numerical tests compare these controllers across optimality, training time, testing time, and robustness, providing a comprehensive evaluation of their efficacy. RL not only yields optimal solutions quickly but also ensures robustness to shifts in customer behavior, such as changes in demand distribution. However, as expected, training the RL agent is more time-consuming than MPC.