Beyond Rewards: a Hierarchical Perspective on Offline Multiagent Behavioral Analysis
Not on the map. Why?
Abstract (arXiv)
Each year, expert-level performance is attained in increasingly-complex multiagent domains, where notable examples include Go, Poker, and StarCraft II. This rapid progression is accompanied by a commensurate need to better understand how such agents attain this performance, to enable their safe deployment, identify limitations, and reveal potential means of improving them. In this paper we take a step back from performance-focused multiagent learning, and instead turn our attention towards agent behavior analysis. We introduce a model-agnostic method for discovery of behavior clusters in multiagent domains, using variational inference to learn a hierarchy of behaviors at the joint and local agent levels. Our framework makes no assumption about agents' underlying learning algorithms, does not require access to their latent states or policies, and is trained using only offline observational data. We illustrate the effectiveness of our method for enabling the coupled understanding of behaviors at the joint and local agent level, detection of behavior changepoints throughout training, discovery of core behavioral concepts, demonstrate the approach's scalability to a high-dimensional multiagent MuJoCo control domain, and also illustrate that the approach can disentangle previously-trained policies in OpenAI's hide-and-seek domain.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 60 of its 92 references that are in the map. First posted on arXiv 17 Jun 2022.
Builds on
60 of 92 references.
References · 60
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
- 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
- 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
- 2020A Primer in BERTology: What we know about how BERT workscited by 815
- 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
- 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 637
- 2019Learning Latent Plans from Playcited by 316
- 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 313
- 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 313
- 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 264
38 more
- 2020Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Imagescited by 264
- 2020Understanding the Role of Individual Units in a Deep Neural Networkcited by 255
- 2019On Completeness-aware Concept-Based Explanations in Deep Neural Networkscited by 239
- 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 223
- 2016Variational Deep Embedding: An Unsupervised and Generative Approach to Clusteringcited by 220
- 2018Programmatically Interpretable Reinforcement Learningcited by 193
- 2019Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishingcited by 191
- 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 166
- 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
- 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 138
- 2020RODE: Learning Roles to Decompose Multi-Agent Taskscited by 125
- 2021Acquisition of Chess Knowledge in AlphaZerocited by 123
- 2018Variational Option Discovery Algorithmscited by 122
- 2020Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skillscited by 122
- 2017Emergent Complexity via Multi-Agent Competitioncited by 120
- 2020Effective Diversity in Population Based Reinforcement Learningcited by 119
- 2017Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observationscited by 100
- 2020Explainability in Deep Reinforcement Learningcited by 97
- 2021Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharingcited by 87
- 2021From Motor Control to Team Play in Simulated Humanoid Footballcited by 87
- 2021Bottom-Up Skill Discovery from Unsegmented Demonstrations for Long-Horizon Robot Manipulationcited by 83
- 2016Graying the black box: Understanding DQNscited by 68
- 2018Latent Space Policies for Hierarchical Reinforcement Learningcited by 67
- 2021The Option Keyboard: Combining Skills in Reinforcement Learningcited by 67
- 2018Diversity-Driven Exploration Strategy for Deep Reinforcement Learningcited by 65
- 2018Emergent Communication through Negotiationcited by 62
- 2019Emergent Coordination Through Competitioncited by 57
- 2020Learning Robot Skills with Temporal Variational Inferencecited by 54
- 2019Interestingness Elements for Explainable Reinforcement Learning: Understanding Agents' Capabilities and Limitationscited by 50
- 2019Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discoverycited by 43
- 2017OptionGAN: Learning Joint Reward-Policy Options using Generative Adversarial Inverse Reinforcement Learningcited by 40
- 2021DISSECT: Disentangled Simultaneous Explanations via Concept Traversalscited by 37
- 2021Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomizationcited by 35
- 2020Relative Variational Intrinsic Controlcited by 34
- 2021RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learningcited by 31
- 2018Learning Self-Imitating Diverse Policiescited by 29
- 2021Unsupervised Skill Discovery with Bottleneck Option Learningcited by 28
- 2021Learning Transferable Motor Skills with Hierarchical Latent Mixture Policiescited by 26
50 most cited of 60 All 60 (JSON)
Built on by
Citing papers per year: 2023: 3.
3 papers.