# A Learning-based Optimal Market Bidding Strategy for Price-Maker Energy Storage

arXiv: https://arxiv.org/abs/2106.02396 · HTML: https://mlsanity.com/paper/2106.02396/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 1 of its 27 references that are in the map.
- First posted on arXiv 4 Jun 2021.

## Builds on (most cited first, 1 of 1)

- [A Comparison of Policies on the Participation of Storage in U.S. Frequency Regulation Markets](https://mlsanity.com/paper/1602.04420/) (2016)

## Built on by (most cited first, 4 of 4)

- [Safe Reinforcement Learning for Strategic Bidding of Virtual Power Plants in Day-Ahead Markets](https://mlsanity.com/paper/2307.05812/) (2023)
- [Temporal-Aware Deep Reinforcement Learning for Energy Storage Bidding in Energy and Contingency Reserve Markets](https://mlsanity.com/paper/2402.19110/) (2024)
- [Deep Reinforcement Learning for Community Battery Scheduling under Uncertainties of Load, PV Generation, and Energy Prices](https://mlsanity.com/paper/2312.03008/) (2023)
- [Attentive Convolutional Deep Reinforcement Learning for Optimizing Solar-Storage Systems in Real-Time Electricity Markets](https://mlsanity.com/paper/2401.15853/) (2024)

## Abstract (arXiv)

Load serving entities with storage units reach sizes and performances that can significantly impact clearing prices in electricity markets. Nevertheless, price endogeneity is rarely considered in storage bidding strategies and modeling the electricity market is a challenging task. Meanwhile, model-free reinforcement learning such as the Actor-Critic are becoming increasingly popular for designing energy system controllers. Yet implementation frequently requires lengthy, data-intense, and unsafe trial-and-error training. To fill these gaps, we implement an online Supervised Actor-Critic (SAC) algorithm, supervised with a model-based controller -- Model Predictive Control (MPC). The energy storage agent is trained with this algorithm to optimally bid while learning and adjusting to its impact on the market clearing prices. We compare the supervised Actor-Critic algorithm with the MPC algorithm as a supervisor, finding that the former reaps higher profits via learning. Our contribution, thus, is an online and safe SAC algorithm that outperforms the current model-based state-of-the-art.