# Renewable energy integration and microgrid energy trading using multi-agent deep reinforcement learning

arXiv: https://arxiv.org/abs/2111.10898 · HTML: https://mlsanity.com/paper/2111.10898/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 55 references that are in the map.
- First posted on arXiv 21 Nov 2021.

## Builds on (most cited first, 10 of 10)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Learning to Communicate with Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1605.06676/) (2016)
- [Distributed Distributional Deterministic Policy Gradients](https://mlsanity.com/paper/1804.08617/) (2018)
- [Multiagent Cooperation and Competition with Deep Reinforcement Learning](https://mlsanity.com/paper/1511.08779/) (2015)
- [Parameter Space Noise for Exploration](https://mlsanity.com/paper/1706.01905/) (2017)
- [Data-driven battery operation for energy arbitrage using rainbow deep reinforcement learning](https://mlsanity.com/paper/2106.06061/) (2021)

## Built on by (most cited first, 8 of 8)

- [A Holistic Power Optimization Approach for Microgrid Control Based on Deep Reinforcement Learning](https://mlsanity.com/paper/2403.01013/) (2024)
- [Reinforcement Learning for Sustainable Energy: A Survey](https://mlsanity.com/paper/2407.18597/) (2024)
- [Augmenting the action space with conventions to improve multi-agent cooperation in Hanabi](https://mlsanity.com/paper/2412.06333/) (2024)
- [Green Energy Management for Sustainable Data Centers Using Deep Reinforcement Learning](https://mlsanity.com/paper/2507.21153/) (2025)
- [Battery and Hydrogen Energy Storage Control in a Smart Energy Network with Flexible Energy Demand using Deep Reinforcement Learning](https://mlsanity.com/paper/2208.12779/) (2022)
- [Credit-cognisant reinforcement learning for multi-agent cooperation](https://mlsanity.com/paper/2211.10100/) (2022)
- [A New Error Temporal Difference Algorithm for Deep Reinforcement Learning in Microgrid Optimization](https://mlsanity.com/paper/2511.18093/) (2025)
- [Incentive-Aligned Vehicle-to-Vehicle Energy Trading via Nash-Integrated Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2605.22363/) (2026)

## Abstract (arXiv)

In this paper, multi-agent reinforcement learning is used to control a hybrid energy storage system working collaboratively to reduce the energy costs of a microgrid through maximising the value of renewable energy and trading. The agents must learn to control three different types of energy storage system suited for short, medium, and long-term storage under fluctuating demand, dynamic wholesale energy prices, and unpredictable renewable energy generation. Two case studies are considered: the first looking at how the energy storage systems can better integrate renewable energy generation under dynamic pricing, and the second with how those same agents can be used alongside an aggregator agent to sell energy to self-interested external microgrids looking to reduce their own energy bills. This work found that the centralised learning with decentralised execution of the multi-agent deep deterministic policy gradient and its state-of-the-art variants allowed the multi-agent methods to perform significantly better than the control from a single global agent. It was also found that using separate reward functions in the multi-agent approach performed much better than using a single control agent. Being able to trade with the other microgrids, rather than just selling back to the utility grid, also was found to greatly increase the grid's savings.