# Data-driven battery operation for energy arbitrage using rainbow deep reinforcement learning

arXiv: https://arxiv.org/abs/2106.06061 · HTML: https://mlsanity.com/paper/2106.06061/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 44 references that are in the map.
- First posted on arXiv 10 Jun 2021.

## Builds on (most cited first, 10 of 10)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [On-line Building Energy Optimization using Deep Reinforcement Learning](https://mlsanity.com/paper/1707.05878/) (2017)
- [Model Predictive Control for Distributed Microgrid Battery Energy Storage Systems](https://mlsanity.com/paper/1702.04699/) (2017)
- [Lifelong Control of Off-grid Microgrid with Model Based Reinforcement Learning](https://mlsanity.com/paper/2005.08006/) (2020)

## Built on by (most cited first, 9 of 9)

- [Renewable energy integration and microgrid energy trading using multi-agent deep reinforcement learning](https://mlsanity.com/paper/2111.10898/) (2021)
- [Sibyl: Adaptive and Extensible Data Placement in Hybrid Storage Systems Using Online Reinforcement Learning](https://mlsanity.com/paper/2205.07394/) (2022)
- [Enhancing Battery Storage Energy Arbitrage with Deep Reinforcement Learning and Time-Series Forecasting](https://mlsanity.com/paper/2410.20005/) (2024)
- [Augmenting the action space with conventions to improve multi-agent cooperation in Hanabi](https://mlsanity.com/paper/2412.06333/) (2024)
- [Harmonia: Enhancing Data Placement and Migration in Hybrid Storage Systems via Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2503.20507/) (2025)
- [Applications of Reinforcement Learning in Deregulated Power Market: A Comprehensive Review](https://mlsanity.com/paper/2205.08369/) (2022)
- [Credit-cognisant reinforcement learning for multi-agent cooperation](https://mlsanity.com/paper/2211.10100/) (2022)
- [Learning a local trading strategy: deep reinforcement learning for grid-scale renewable energy integration](https://mlsanity.com/paper/2411.15422/) (2024)
- [The Economic Dispatch of Power-to-Gas Systems with Deep Reinforcement Learning:Tackling the Challenge of Delayed Rewards with Long-Term Energy Storage](https://mlsanity.com/paper/2506.06484/) (2025)

## Abstract (arXiv)

As the world seeks to become more sustainable, intelligent solutions are needed to increase the penetration of renewable energy. In this paper, the model-free deep reinforcement learning algorithm Rainbow Deep Q-Networks is used to control a battery in a small microgrid to perform energy arbitrage and more efficiently utilise solar and wind energy sources. The grid operates with its own demand and renewable generation based on a dataset collected at Keele University, as well as using dynamic energy pricing from a real wholesale energy market. Four scenarios are tested including using demand and price forecasting produced with local weather data. The algorithm and its subcomponents are evaluated against two continuous control benchmarks with Rainbow able to outperform all other method. This research shows the importance of using the distributional approach for reinforcement learning when working with complex environments and reward functions, as well as how it can be used to visualise and contextualise the agent's behaviour for real-world applications.