# A Deep Reinforcement Learning Framework for Continuous Intraday Market Bidding

arXiv: https://arxiv.org/abs/2004.05940 · HTML: https://mlsanity.com/paper/2004.05940/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Apr 2020.

## Built on by (most cited first, 15 of 15)

- [Reinforcement Learning in Practice: Opportunities and Challenges](https://mlsanity.com/paper/2202.11296/) (2022)
- [Distributional Reinforcement Learning-based Energy Arbitrage Strategies in Imbalance Settlement Mechanism](https://mlsanity.com/paper/2401.00015/) (2023)
- [Advancing Investment Frontiers: Industry-grade Deep Reinforcement Learning for Portfolio Optimization](https://mlsanity.com/paper/2403.07916/) (2024)
- [Bridging the Performance Gap Between Target-Free and Target-Based Reinforcement Learning](https://mlsanity.com/paper/2506.04398/) (2025)
- [Behind the Myth of Exploration in Policy Gradients](https://mlsanity.com/paper/2402.00162/) (2024)
- [Off-Policy Maximum Entropy RL with Future State and Action Visitation Measures](https://mlsanity.com/paper/2412.06655/) (2024)
- [Model Predictive Control-Guided Reinforcement Learning for Implicit Balancing](https://mlsanity.com/paper/2510.04868/) (2025)
- [Application of deep reinforcement learning for Indian stock trading automation](https://mlsanity.com/paper/2106.16088/) (2021)
- [Joint Bidding on Intraday and Frequency Containment Reserve Markets](https://mlsanity.com/paper/2510.03209/) (2025)
- [Gym-TORAX: Open-source software for integrating reinforcement learning with plasma control simulators in tokamak research](https://mlsanity.com/paper/2510.11283/) (2025)
- [TradeR: Practical Deep Hierarchical Reinforcement Learning for Trade Execution](https://mlsanity.com/paper/2104.00620/) (2021)
- [Parameter-free Reduction of the Estimation Bias in Deep Reinforcement Learning for Deterministic Policy Gradients](https://mlsanity.com/paper/2109.11788/) (2021)
- [Domain-adapted Learning and Imitation: DRL for Power Arbitrage](https://mlsanity.com/paper/2301.08360/) (2023)
- [Feature-driven reinforcement learning for photovoltaic in continuous intraday trading](https://mlsanity.com/paper/2510.16021/) (2025)
- [Maximum-Entropy Exploration with Future State-Action Visitation Measures](https://mlsanity.com/paper/2603.18965/) (2026)

## Abstract (arXiv)

The large integration of variable energy resources is expected to shift a large part of the energy exchanges closer to real-time, where more accurate forecasts are available. In this context, the short-term electricity markets and in particular the intraday market are considered a suitable trading floor for these exchanges to occur. A key component for the successful renewable energy sources integration is the usage of energy storage. In this paper, we propose a novel modelling framework for the strategic participation of energy storage in the European continuous intraday market where exchanges occur through a centralized order book. The goal of the storage device operator is the maximization of the profits received over the entire trading horizon, while taking into account the operational constraints of the unit. The sequential decision-making problem of trading in the intraday market is modelled as a Markov Decision Process. An asynchronous distributed version of the fitted Q iteration algorithm is chosen for solving this problem due to its sample efficiency. The large and variable number of the existing orders in the order book motivates the use of high-level actions and an alternative state representation. Historical data are used for the generation of a large number of artificial trajectories in order to address exploration issues during the learning process. The resulting policy is back-tested and compared against a benchmark strategy that is the current industrial standard. Results indicate that the agent converges to a policy that achieves in average higher total revenues than the benchmark strategy.