mlsanityData to 7 Oct 2026

Learning Multi-agent Skills for Tabular Reinforcement Learning using Factor Graphs

arXiv 2201.08227 ↗First posted 20 Jan 2022cs.MA cs.AI cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Covering skill (a.k.a., option) discovery has been developed to improve the exploration of reinforcement learning in single-agent scenarios with sparse reward signals, through connecting the most distant states in the embedding space provided by the Fiedler vector of the state transition graph. However, these option discovery methods cannot be directly extended to multi-agent scenarios, since the joint state space grows exponentially with the number of agents in the system. Thus, existing researches on adopting options in multi-agent scenarios still rely on single-agent option discovery and fail to directly discover the joint options that can improve the connectivity of the joint state space of agents. In this paper, we show that it is indeed possible to directly compute multi-agent options with collaborative exploratory behaviors among the agents, while still enjoying the ease of decomposition. Our key idea is to approximate the joint state space as a Kronecker graph -- the Kronecker product of individual agents' state transition graphs, based on which we can directly estimate the Fiedler vector of the joint state space using the Laplacian spectrum of individual agents' transition graphs. This decomposition enables us to efficiently construct multi-agent joint options by encouraging agents to connect the sub-goal joint states which are corresponding to the minimum or maximum values of the estimated joint Fiedler vector. The evaluation based on multi-agent collaborative tasks shows that the proposed algorithm can successfully identify multi-agent options, and significantly outperforms prior works using single-agent options or no options, in terms of both faster exploration and higher cumulative rewards.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 21 of its 55 references that are in the map. First posted on arXiv 20 Jan 2022.

Builds on

21 of 55 references.

References · 21

  1. 2015Continuous control with deep reinforcement learningcited by 3,279
  2. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  3. 2017Counterfactual Multi-Agent Policy Gradientscited by 733
  4. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328
  5. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
  6. 2020Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 197
  7. 2019MAVEN: Multi-Agent Variational Explorationcited by 181
  8. 2017A Laplacian Framework for Option Discovery in Reinforcement Learningcited by 108
  9. 2017Eigenoption Discovery through the Deep Successor Representationcited by 85
  10. 2018The Laplacian in RL: Learning Representations with Efficient Approximationscited by 83
  11. 2017When Waiting is not an Option : Learning Options with a Deliberation Costcited by 57
  12. 2019Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discoverycited by 42
9 more
  1. 2019Discovering Options for Exploration by Minimizing Cover Timecited by 30
  2. 2016Playing Atari Games with Deep Reinforcement Learning and Human Checkpoint Replaycited by 22
  3. 2021Towards Better Laplacian Representation in Reinforcement Learning with Generalized Graph Drawingcited by 22
  4. 2017Exploration--Exploitation in MDPs with Optionscited by 19
  5. 2018Finding Options that Minimize Planning Timecited by 17
  6. 2019Option-Critic in Cooperative Multi-agent Systemscited by 17
  7. 2016Adaptive Skills, Adaptive Partitions (ASAP)cited by 11
  8. 2015Estimation of Laplacian spectra of direct and strong product graphscited by 4
  9. 2021Another estimation of Laplacian spectrum of the Kronecker product of graphscited by 3

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 1.

4 papers.

  1. 2023Scalable Multi-agent Covering Option Discovery based on Kronecker Graphscited by 9
  2. 2022A Unified Algorithm Framework for Unsupervised Discovery of Skills based on Determinantal Point Processcited by 7
  3. 2023Hierarchical Deep Counterfactual Regret Minimizationcited by 4
  4. 2024Enhancing the Resilience of Graph Neural Networks to Topological Perturbations in Sparse Graphscited by 3