mlsanityData to 7 Oct 2026

Rainbow Delay Compensation: A Multi-Agent Reinforcement Learning Framework for Mitigating Delayed Observation

arXiv 2505.03586 ↗First posted 6 May 2025cs.MA cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

In real-world multi-agent systems (MASs), observation delays are ubiquitous, preventing agents from making decisions based on the environment's true state. An individual agent's local observation typically comprises multiple components from other agents or dynamic entities within the environment. These discrete observation components with varying delay characteristics pose significant challenges for multi-agent reinforcement learning (MARL). In this paper, we first formulate the decentralized stochastic individual delay partially observable Markov decision process (DSID-POMDP) by extending the standard Dec-POMDP. We then propose the Rainbow Delay Compensation (RDC), a MARL training framework for addressing stochastic individual delays, along with recommended implementations for its constituent modules. We implement the DSID-POMDP's observation generation pattern using standard MARL benchmarks, including MPE and SMAC. Experiments demonstrate that baseline MARL methods suffer severe performance degradation under fixed and unfixed delays. The RDC-enhanced approach mitigates this issue, remarkably achieving ideal delay-free performance in certain delay scenarios while maintaining generalizability. Our work provides a novel perspective on multi-agent delayed observation problems and offers an effective solution framework. The source code is available at https://github.com/linkjoker1006/RDC-pymarl.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 26 of its 41 references that are in the map. First posted on arXiv 6 May 2025.

Builds on

26 of 41 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  6. 2019The StarCraft Multi-Agent Challengecited by 589
  7. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 555
  8. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 490
  9. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 475
  10. 2019Google Research Football: A Novel Reinforcement Learning Environmentcited by 257
  11. 2020Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Taskscited by 207
  12. 2022A Survey of Multi-Agent Deep Reinforcement Learning with Communicationcited by 103
14 more
  1. 2021Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learningcited by 68
  2. 2020Reinforcement Learning with Random Delayscited by 56
  3. 2023Robust Multi-Agent Reinforcement Learning with State Uncertaintycited by 35
  4. 2020Delay-Aware Model-Based Reinforcement Learning for Continuous Controlcited by 33
  5. 2018At Human Speed: Deep Reinforcement Learning with Action Delaycited by 24
  6. 2020Delay-Aware Multi-Agent Reinforcement Learning for Cooperative and Competitive Environmentscited by 21
  7. 2022Delayed Reinforcement Learning by Imitationcited by 15
  8. 2022Multi-Agent Reinforcement Learning with Reward Delayscited by 9
  9. 2022DACOM: Learning Delay-Aware Communication for Multi-Agent Reinforcement Learningcited by 9
  10. 2023Emergent Communication in Multi-Agent Reinforcement Learning for Future Wireless Networkscited by 9
  11. 2023A Reduction-based Framework for Sequential Decision Making with Delayed Feedbackcited by 7
  12. 2023Optimizing Trading Strategies in Quantitative Markets using Multi-Agent Reinforcement Learningcited by 7
  13. 2024Delay-Aware Multi-Agent Reinforcement Learning for Cooperative Adaptive Cruise Control with Model-based Stability Enhancementcited by 4
  14. 2024QTypeMix: Enhancing Multi-Agent Cooperative Strategies through Heterogeneous and Homogeneous Value Decompositioncited by 3

Built on by

Not cited yet.