Off-Beat Multi-Agent Reinforcement Learning
Not on the map. Why?
Abstract (arXiv)
We investigate model-free multi-agent reinforcement learning (MARL) in environments where off-beat actions are prevalent, i.e., all actions have pre-set execution durations. During execution durations, the environment changes are influenced by, but not synchronised with, action execution. Such a setting is ubiquitous in many real-world problems. However, most MARL methods assume actions are executed immediately after inference, which is often unrealistic and can lead to catastrophic failure for multi-agent coordination with off-beat actions. In order to fill this gap, we develop an algorithmic framework for MARL with off-beat actions. We then propose a novel episodic memory, LeGEM, for model-free MARL algorithms. LeGEM builds agents' episodic memories by utilizing agents' individual experiences. It boosts multi-agent learning by addressing the challenging temporal credit assignment problem raised by the off-beat actions via our novel reward redistribution scheme, alleviating the issue of non-Markovian reward. We evaluate LeGEM on various multi-agent scenarios with off-beat actions, including Stag-Hunter Game, Quarry Game, Afforestation Game, and StarCraft II micromanagement tasks. Empirical results show that LeGEM significantly boosts multi-agent coordination and achieves leading performance and improved sample efficiency.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 49 of its 76 references that are in the map. First posted on arXiv 27 May 2022.
Builds on
49 of 76 references.
References · 49
- 2017Attention Is All You Needcited by 60,361
- 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
- 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
- 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
- 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
- 2017Counterfactual Multi-Agent Policy Gradientscited by 751
- 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 708
- 2019The StarCraft Multi-Agent Challengecited by 597
- 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 503
- 2016The Option-Critic Architecturecited by 448
- 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 379
- 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 309
37 more
- 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 270
- 2021Trust Region Policy Optimisation in Multi-Agent Reinforcement Learningcited by 186
- 2019MAVEN: Multi-Agent Variational Explorationcited by 185
- 2015Multiagent Cooperation and Competition with Deep Reinforcement Learningcited by 184
- 2018RUDDER: Return Decomposition for Delayed Rewardscited by 173
- 2017Neural Episodic Controlcited by 151
- 2018Learning Scheduling Algorithms for Data Processing Clusterscited by 134
- 2020SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Drivingcited by 121
- 2021Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Potcited by 99
- 2020Off-Policy Multi-Agent Decomposed Policy Gradientscited by 98
- 2021Reward is enough for convex MDPscited by 84
- 2019Learning Efficient Multi-agent Communication: An Information Bottleneck Approachcited by 61
- 2020Reinforcement Learning with Random Delayscited by 59
- 2018Optimizing Agent Behavior over Long Time Scales by Transporting Valuecited by 58
- 2021Episodic Multi-agent Reinforcement Learning with Curiosity-Driven Explorationcited by 49
- 2017Guided Deep Reinforcement Learning for Swarm Systemscited by 44
- 2019Real-Time Reinforcement Learningcited by 40
- 2021Towards mental time travel: a hierarchical memory for reinforcement learning agentscited by 40
- 2021Collective Intelligence for Deep Learning: A Survey of Recent Developmentscited by 34
- 2021Learning Long-Term Reward Redistribution via Randomized Return Decompositioncited by 33
- 2020Thinking While Moving: Deep Reinforcement Learning with Concurrent Controlcited by 31
- 2021Acting in Delayed Environments with Non-Stationary Markov Policiescited by 31
- 2021RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agentscited by 31
- 2020Learning Guidance Rewards with Trajectory-space Smoothingcited by 30
- 2020Expected Eligibility Tracescited by 27
- 2021Synthetic Returns for Long-Term Credit Assignmentcited by 27
- 2021Off-Policy Reinforcement Learning with Delayed Rewardscited by 25
- 2021Generalizable Episodic Memory for Deep Reinforcement Learningcited by 23
- 2020Delay-Aware Multi-Agent Reinforcement Learning for Cooperative and Competitive Environmentscited by 22
- 2022Asynchronous Reinforcement Learning for Real-Time Control of Physical Robotscited by 22
- 2019Action Semantics Network: Considering the Effects of Actions in Multiagent Systemscited by 19
- 2021Regularized Softmax Deep Multi-Agent Q-Learningcited by 19
- 2020Macro-Action-Based Deep Multi-Agent Reinforcement Learningcited by 18
- 2020Reward Propagation Using Graph Convolutional Networkscited by 15
- 2015Decentralized Control of Partially Observable Markov Decision Processes using Belief Space Macro-actionscited by 11
- 2019On Memory Mechanism in Multi-Agent Reinforcement Learningcited by 1
- 2021State-based Episodic Memory for Multi-Agent Reinforcement Learningcited by 1
Built on by
Citing papers per year: 2022: 1.
1 paper.