# How to Use Reinforcement Learning to Facilitate Future Electricity Market Design? Part 1: A Paradigmatic Theory

arXiv: https://arxiv.org/abs/2305.02485 · HTML: https://mlsanity.com/paper/2305.02485/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 46 references that are in the map.
- First posted on arXiv 4 May 2023.

## Builds on (most cited first, 6 of 6)

- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Multi-Agent Trust Region Policy Optimization](https://mlsanity.com/paper/2010.07916/) (2020)
- [Learning from Past Bids to Participate Strategically in Day-Ahead Electricity Markets](https://mlsanity.com/paper/1811.06113/) (2018)
- [Optimal Joint Bidding and Pricing of Profit-seeking Load Serving Entity](https://mlsanity.com/paper/1708.04645/) (2017)
- [Active Distribution Grids offering Ancillary Services in Islanded and Grid-connected Mode](https://mlsanity.com/paper/1906.04804/) (2019)
- [Assigning Shadow Prices to Synthetic Inertia and Frequency Response Reserves from Renewable Energy Sources](https://mlsanity.com/paper/2208.04869/) (2022)

## Abstract (arXiv)

In face of the pressing need of decarbonization in the power sector, the re-design of electricity market is necessary as a Marco-level approach to accommodate the high penetration of renewable generations, and to achieve power system operation security, economic efficiency, and environmental friendliness. However, existing market design methodologies suffer from the lack of coordination among energy spot market (ESM), ancillary service market (ASM) and financial market (FM), i.e., the "joint market", and the lack of reliable simulation-based verification. To tackle these deficiencies, this two-part paper develops a paradigmatic theory and detailed methods of the joint market design using reinforcement-learning (RL)-based simulation. In Part 1, the theory and framework of this novel market design philosophy are proposed. First, the controversial market design options while designing the joint market are summarized as the targeted research questions. Second, the Markov game model is developed to describe the bidding game in the joint market, incorporating the market design options to be determined. Third, a framework of deploying multiple types of RL algorithms to simulate the market model is developed. Finally, several market operation performance indicators are proposed to validate the market design based on the simulation results.