# MARLEM: A Multi-Agent Reinforcement Learning Simulation Framework for Implicit Cooperation in Decentralized Local Energy Markets

arXiv: https://arxiv.org/abs/2602.16063 · HTML: https://mlsanity.com/paper/2602.16063/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 56 references that are in the map.
- First posted on arXiv 17 Feb 2026.

## Builds on (most cited first, 20 of 20)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The StarCraft Multi-Agent Challenge](https://mlsanity.com/paper/1902.04043/) (2019)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks](https://mlsanity.com/paper/2006.07869/) (2020)
- [A Survey of Multi-Agent Deep Reinforcement Learning with Communication](https://mlsanity.com/paper/2203.08975/) (2022)
- [pandapower - an Open Source Python Tool for Convenient Modeling, Analysis and Optimization of Electric Power Systems](https://mlsanity.com/paper/1709.06743/) (2017)
- [Deep Implicit Coordination Graphs for Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2006.11438/) (2020)
- [Learning to run a Power Network Challenge: a Retrospective Analysis](https://mlsanity.com/paper/2103.03104/) (2021)
- [CityLearn: Standardizing Research in Multi-Agent Reinforcement Learning for Demand Response and Urban Energy Management](https://mlsanity.com/paper/2012.10504/) (2020)
- [MARLlib: A Scalable and Efficient Multi-agent Reinforcement Learning Library](https://mlsanity.com/paper/2210.13708/) (2022)
- [Gym-ANM: Reinforcement Learning Environments for Active Network Management Tasks in Electricity Distribution Systems](https://mlsanity.com/paper/2103.07932/) (2021)
- [PowerGridworld: A Framework for Multi-Agent Reinforcement Learning in Power Systems](https://mlsanity.com/paper/2111.05969/) (2021)
- [CityLearn v2: Energy-flexible, resilient, occupant-centric, and carbon-aware management of grid-interactive communities](https://mlsanity.com/paper/2405.03848/) (2024)
- [Learning to Communicate Implicitly By Actions](https://mlsanity.com/paper/1810.04444/) (2018)
- [Centralised rehearsal of decentralised cooperation: Multi-agent reinforcement learning for the scalable coordination of residential energy flexibility](https://mlsanity.com/paper/2305.18875/) (2023)
- [EVLearn: Extending the CityLearn Framework with Electric Vehicle Simulation](https://mlsanity.com/paper/2404.06521/) (2024)
- [Decentralized P2P Energy Trading under Network Constraints in a Low-Voltage Network](https://mlsanity.com/paper/1809.06976/) (2018)
- [MARL-iDR: Multi-Agent Reinforcement Learning for Incentive-based Residential Demand Response](https://mlsanity.com/paper/2304.04086/) (2023)
- [RL-ADN: A High-Performance Deep Reinforcement Learning Environment for Optimal Energy Storage Systems Dispatch in Active Distribution Networks](https://mlsanity.com/paper/2408.03685/) (2024)
- [Multi-Agent Reinforcement Learning for Energy Networks: Computational Challenges, Progress and Open Problems](https://mlsanity.com/paper/2404.15583/) (2024)

## Abstract (arXiv)

This paper introduces a novel, open-source MARL simulation framework for studying implicit cooperation in LEMs, modeled as a decentralized partially observable Markov decision process and implemented as a Gymnasium environment for MARL. Our framework features a modular market platform with plug-and-play clearing mechanisms, physically constrained agent models (including battery storage), a realistic grid network, and a comprehensive analytics suite to evaluate emergent coordination. The main contribution is a novel method to foster implicit cooperation, where agents' observations and rewards are enhanced with system-level key performance indicators to enable them to independently learn strategies that benefit the entire system and aim for collectively beneficial outcomes without explicit communication. Through representative case studies (available in a dedicated GitHub repository in https://github.com/salazarna/marlem, we show the framework's ability to analyze how different market configurations (such as varying storage deployment) impact system performance. This illustrates its potential to facilitate emergent coordination, improve market efficiency, and strengthen grid stability. The proposed simulation framework is a flexible, extensible, and reproducible tool for researchers and practitioners to design, test, and validate strategies for future intelligent, decentralized energy systems.