# Distributional Reinforcement Learning-based Energy Arbitrage Strategies in Imbalance Settlement Mechanism

arXiv: https://arxiv.org/abs/2401.00015 · HTML: https://mlsanity.com/paper/2401.00015/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 61 references that are in the map.
- First posted on arXiv 23 Dec 2023.

## Builds on (most cited first, 12 of 12)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors](https://mlsanity.com/paper/2001.02811/) (2020)
- [Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policies](https://mlsanity.com/paper/2111.02552/) (2021)
- [A Deep Reinforcement Learning Framework for Continuous Intraday Market Bidding](https://mlsanity.com/paper/2004.05940/) (2020)
- [Definition and evaluation of model-free coordination of electrical vehicle charging with reinforcement learning](https://mlsanity.com/paper/1809.10679/) (2018)
- [Risk-Sensitive Policy with Distributional Reinforcement Learning](https://mlsanity.com/paper/2212.14743/) (2022)
- [Energy Storage Price Arbitrage via Opportunity Value Function Prediction](https://mlsanity.com/paper/2211.07797/) (2022)
- [Life cycle economic viability analysis of battery storage in electricity market](https://mlsanity.com/paper/2305.15079/) (2023)
- [Combined Stochastic Optimization of Frequency Control and Self-Consumption with a Battery](https://mlsanity.com/paper/1710.08331/) (2017)
- [Techno-Economic Analysis and Optimal Control of Battery Storage for Frequency Control Services, Applied to the German Market](https://mlsanity.com/paper/1903.04251/) (2019)

## Built on by (most cited first, 5 of 5)

- [Model Predictive Control-Guided Reinforcement Learning for Implicit Balancing](https://mlsanity.com/paper/2510.04868/) (2025)
- [Control Policy Correction Framework for Reinforcement Learning-based Energy Arbitrage Strategies](https://mlsanity.com/paper/2404.18821/) (2024)
- [Interpretable reinforcement learning for heat pump control through asymmetric differentiable decision trees](https://mlsanity.com/paper/2506.01641/) (2025)
- [Neural Network-Assisted Model Predictive Control for Implicit Balancing](https://mlsanity.com/paper/2604.01805/) (2026)
- [Multi-market value-stacking: Battery control for combined imbalance participation and non-uniform FCR bidding](https://mlsanity.com/paper/2605.23964/) (2026)

## Abstract (arXiv)

Growth in the penetration of renewable energy sources makes supply more uncertain and leads to an increase in the system imbalance. This trend, together with the single imbalance pricing, opens an opportunity for balance responsible parties (BRPs) to perform energy arbitrage in the imbalance settlement mechanism. To this end, we propose a battery control framework based on distributional reinforcement learning (DRL). Our proposed control framework takes a risk-sensitive perspective, allowing BRPs to adjust their risk preferences: we aim to optimize a weighted sum of the arbitrage profit and a risk measure while constraining the daily number of cycles for the battery. We assess the performance of our proposed control framework using the Belgian imbalance prices of 2022 and compare two state-of-the-art RL methods, deep Q learning and soft actor-critic. Results reveal that the distributional soft actor-critic method can outperform other methods. Moreover, we note that our fully risk-averse agent appropriately learns to hedge against the risk related to the unknown imbalance price by (dis)charging the battery only when the agent is more certain about the price.