# Characterizing MARL for Energy Control: A Multi-KPI Benchmark on the CityLearn Environment

arXiv: https://arxiv.org/abs/2602.19223 · HTML: https://mlsanity.com/paper/2602.19223/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 34 references that are in the map.
- First posted on arXiv 22 Feb 2026.

## Builds on (most cited first, 17 of 17)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Multi-Agent Reinforcement Learning is a Sequence Modeling Problem](https://mlsanity.com/paper/2205.14953/) (2022)
- [Memory-based control with recurrent neural networks](https://mlsanity.com/paper/1512.04455/) (2015)
- [Measuring the Reliability of Reinforcement Learning Algorithms](https://mlsanity.com/paper/1912.05663/) (2019)
- [Towards a Standardised Performance Evaluation Protocol for Cooperative MARL](https://mlsanity.com/paper/2209.10485/) (2022)
- [GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms](https://mlsanity.com/paper/1802.05054/) (2018)
- [Podracer architectures for scalable Reinforcement Learning](https://mlsanity.com/paper/2104.06272/) (2021)
- [Multi-Agent Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2408.09675/) (2024)
- [CityLearn: Standardizing Research in Multi-Agent Reinforcement Learning for Demand Response and Urban Energy Management](https://mlsanity.com/paper/2012.10504/) (2020)
- [Multi-Agent Reinforcement Learning for Resources Allocation Optimization: A Survey](https://mlsanity.com/paper/2504.21048/) (2025)
- [Sable: a Performant, Efficient and Scalable Sequence Model for MARL](https://mlsanity.com/paper/2410.01706/) (2024)
- [A Centralised Soft Actor Critic Deep Reinforcement Learning Approach to District Demand Side Management through CityLearn](https://mlsanity.com/paper/2009.10562/) (2020)
- [Winning the CityLearn Challenge: Adaptive Optimization with Evolutionary Search under Trajectory-based Guidance](https://mlsanity.com/paper/2212.01939/) (2022)
- [Efficiently Quantifying Individual Agent Importance in Cooperative MARL](https://mlsanity.com/paper/2312.08466/) (2023)

## Abstract (arXiv)

The optimization of urban energy systems is crucial for the advancement of sustainable and resilient smart cities, which are becoming increasingly complex with multiple decision-making units. To address scalability and coordination concerns, Multi-Agent Reinforcement Learning (MARL) is a promising solution. This paper addresses the imperative need for comprehensive and reliable benchmarking of MARL algorithms on energy management tasks. CityLearn is used as a case study environment because it realistically simulates urban energy systems, incorporates multiple storage systems, and utilizes renewable energy sources. By doing so, our work sets a new standard for evaluation, conducting a comparative study across multiple key performance indicators (KPIs). This approach illuminates the key strengths and weaknesses of various algorithms, moving beyond traditional KPI averaging which often masks critical insights. Our experiments utilize widely accepted baselines such as Proximal Policy Optimization (PPO) and Soft Actor Critic (SAC), and encompass diverse training schemes including Decentralized Training with Decentralized Execution (DTDE) and Centralized Training with Decentralized Execution (CTDE) approaches and different neural network architectures. Our work also proposes novel KPIs that tackle real world implementation challenges such as individual building contribution and battery storage lifetime. Our findings show that DTDE consistently outperforms CTDE in both average and worst-case performance. Additionally, temporal dependency learning improved control on memory dependent KPIs such as ramping and battery usage, contributing to more sustainable battery operation. Results also reveal robustness to agent or resource removal, highlighting both the resilience and decentralizability of the learned policies.