mlsanityData to 13 Oct 2026

Structured Cooperative Multi-Agent Reinforcement Learning: a Bayesian Network Perspective

arXiv 2510.09937 ↗First posted 11 Oct 2025cs.MA cs.LG cs.SYcited by 0

Not on the map. Why?

Abstract (arXiv)

The empirical success of multi-agent reinforcement learning (MARL) has motivated the search for more efficient and scalable algorithms for large scale multi-agent systems. However, existing state-of-the-art algorithms do not fully exploit inter-agent coupling information to develop MARL algorithms. In this paper, we propose a systematic approach to leverage structures in the inter-agent couplings for efficient model-free reinforcement learning. We model the cooperative MARL problem via a Bayesian network and characterize the subset of agents, termed as the value dependency set, whose information is required by each agent to estimate its local action value function exactly. Moreover, we propose a partially decentralized training decentralized execution (P-DTDE) paradigm based on the value dependency set. We theoretically establish that the total variance of our P-DTDE policy gradient estimator is less than the centralized training decentralized execution (CTDE) policy gradient estimator. We derive a multi-agent policy gradient theorem based on the P-DTDE scheme and develop a scalable actor-critic algorithm. We demonstrate the efficiency and scalability of the proposed algorithm on multi-warehouse resource allocation and multi-zone temperature control examples. For dense value dependency sets, we propose an approximation scheme based on truncation of the Bayesian network and empirically show that it achieves a faster convergence than the exact value dependence set for applications with a large number of agents.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 49 of its 75 references that are in the map. First posted on arXiv 11 Oct 2025.

Builds on

49 of 75 references.

References · 49

  1. 2017Graph Attention Networkscited by 8,888
  2. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,525
  3. 2017Counterfactual Multi-Agent Policy Gradientscited by 739
  4. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 700
  5. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 493
  6. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 477
  7. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 374
  8. 2018Actor-Attention-Critic for Multi-Agent Reinforcement Learningcited by 260
  9. 2018Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agentscited by 190
  10. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  11. 2018A Survey and Critique of Multiagent Deep Reinforcement Learningcited by 174
  12. 2018Graph Convolutional Reinforcement Learningcited by 150
37 more
  1. 2020Off-Policy Multi-Agent Decomposed Policy Gradientscited by 98
  2. 2019Deep Coordination Graphscited by 96
  3. 2020Value-Decomposition Multi-Agent Actor-Criticscited by 47
  4. 2019Reducing Overestimation Bias in Multi-Agent Domains Using Double Centralized Criticscited by 46
  5. 2021Settling the Variance of Multi-Agent Policy Gradientscited by 45
  6. 2020R-MADDPG for Partially Observable Environments and Limited Communicationcited by 36
  7. 2022Scalable Multi-Agent Reinforcement Learning through Intelligent Information Aggregationcited by 35
  8. 2023Heterogeneous Multi-Robot Reinforcement Learningcited by 35
  9. 2019Distributed Reinforcement Learning for Decentralized Linear Quadratic Control: A Derivative-Free Policy Optimization Approachcited by 33
  10. 2020Making Sense of Reinforcement Learning and Probabilistic Inferencecited by 30
  11. 2020Towards Understanding Cooperative Multi-Agent Q-Learning with Value Factorizationcited by 26
  12. 2021Unbiased Asymmetric Reinforcement Learning under Partial Observabilitycited by 25
  13. 2019Scalable Reinforcement Learning for Multi-Agent Networked Systemscited by 23
  14. 2018Modelling the Dynamic Joint Policy of Teammates with Attention Multi-agent DDPGcited by 20
  15. 2019Value Propagation for Decentralized Networked Deep Multi-agent Reinforcement Learningcited by 18
  16. 2019Stochastic Optimal Control as Approximate Input Inferencecited by 14
  17. 2020Graph Convolutional Value Decomposition in Multi-Agent Reinforcement Learningcited by 14
  18. 2022Pareto Actor-Critic for Equilibrium Selection in Multi-Agent Reinforcement Learningcited by 14
  19. 2017Parameter Sharing Deep Deterministic Policy Gradient for Cooperative Multi-agent Reinforcement Learningcited by 13
  20. 2023Probabilistic Inference in Reinforcement Learning Done Rightcited by 11
  21. 2020F2A2: Flexible Fully-decentralized Approximate Actor-critic for Cooperative Multi-agent Reinforcement Learningcited by 10
  22. 2019Exploiting Fast Decaying and Locality in Multi-Agent MDP with Tree Dependence Structurecited by 8
  23. 2021Common Information based Approximate State Representations in Multi-Agent Reinforcement Learningcited by 8
  24. 2020Cooperative Multi-Agent Reinforcement Learning with Partial Observationscited by 6
  25. 2022Distributed Multi-Agent Reinforcement Learning Based on Graph-Induced Local Value Functionscited by 6
  26. 2024Overcoming the Curse of Dimensionality in Reinforcement Learning Through Approximate Factorizationcited by 6
  27. 2017Diff-DAC: Distributed Actor-Critic for Average Multitask Deep Reinforcement Learningcited by 5
  28. 2023Multi-agent Reinforcement Learning with Graph Q-Networks for Antenna Tuningcited by 4
  29. 2021Efficient Stochastic Optimal Control through Approximate Bayesian Input Inferencecited by 3
  30. 2024Approximate Global Convergence of Independent Learning in Multi-Agent Systemscited by 3
  31. 2018Multi-Agent Actor-Critic with Generative Cooperative Policy Networkcited by 2
  32. 2023Multi-agent Policy Reciprocity with Theoretical Guaranteecited by 1
  33. 2023Cooperative Multi-Agent Constrained POMDPs: Strong Duality and Primal-Dual Reinforcement Learning with Approximate Information Statescited by 1
  34. 2023Partially Observable Multi-Agent Reinforcement Learning with Information Sharingcited by 1
  35. 2025Convergence and Connectivity: Dynamics of Multi-Agent Q-Learning in Random Networkscited by 1
  36. 2025Exploiting inter-agent coupling information for efficient reinforcement learning of cooperative LQRcited by 1
  37. 2025Approximate constrained stochastic optimal control via parameterized input inferencecited by 1

Built on by

Not cited yet.