# Energy Storage Arbitrage in Real-Time Markets via Reinforcement Learning

arXiv: https://arxiv.org/abs/1711.03127 · HTML: https://mlsanity.com/paper/1711.03127/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Nov 2017.

## Built on by (most cited first, 14 of 14)

- [Proximal Policy Optimization Based Reinforcement Learning for Joint Bidding in Energy and Frequency Regulation Markets](https://mlsanity.com/paper/2212.06551/) (2022)
- [A Reinforcement Learning Approach for the Continuous Electricity Market of Germany: Trading from the Perspective of a Wind Park Operator](https://mlsanity.com/paper/2111.13609/) (2021)
- [Energy Storage Price Arbitrage via Opportunity Value Function Prediction](https://mlsanity.com/paper/2211.07797/) (2022)
- [Transferable Energy Storage Bidder](https://mlsanity.com/paper/2301.01233/) (2023)
- [Enhancing Battery Storage Energy Arbitrage with Deep Reinforcement Learning and Time-Series Forecasting](https://mlsanity.com/paper/2410.20005/) (2024)
- [Multi-market Energy Optimization with Renewables via Reinforcement Learning](https://mlsanity.com/paper/2306.08147/) (2023)
- [Temporal-Aware Deep Reinforcement Learning for Energy Storage Bidding in Energy and Contingency Reserve Markets](https://mlsanity.com/paper/2402.19110/) (2024)
- [Energy Storage Arbitrage in Two-settlement Markets: A Transformer-Based Approach](https://mlsanity.com/paper/2404.17683/) (2024)
- [Optimizing a domestic battery and solar photovoltaic system with deep reinforcement learning](https://mlsanity.com/paper/2109.05024/) (2021)
- [Deep Reinforcement Learning for Community Battery Scheduling under Uncertainties of Load, PV Generation, and Energy Prices](https://mlsanity.com/paper/2312.03008/) (2023)
- [Learning a local trading strategy: deep reinforcement learning for grid-scale renewable energy integration](https://mlsanity.com/paper/2411.15422/) (2024)
- [Degradation-Aware Frequency Regulation of a Heterogeneous Battery Fleet via Reinforcement Learning](https://mlsanity.com/paper/2601.22865/) (2026)
- [Multi-market value-stacking: Battery control for combined imbalance participation and non-uniform FCR bidding](https://mlsanity.com/paper/2605.23964/) (2026)
- [State Representation Matters in Deep Reinforcement Learning: Application to Energy Trading](https://mlsanity.com/paper/2606.27032/) (2026)

## Abstract (arXiv)

In this paper, we derive a temporal arbitrage policy for storage via reinforcement learning. Real-time price arbitrage is an important source of revenue for storage units, but designing good strategies have proven to be difficult because of the highly uncertain nature of the prices. Instead of current model predictive or dynamic programming approaches, we use reinforcement learning to design an optimal arbitrage policy. This policy is learned through repeated charge and discharge actions performed by the storage unit through updating a value matrix. We design a reward function that does not only reflect the instant profit of charge/discharge decisions but also incorporate the history information. Simulation results demonstrate that our designed reward function leads to significant performance improvement compared with existing algorithms.